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ABSTRACT 


Decision Systems have two components -- a management decision maker 
and a decision support system (DSS). This dissertation evaluates 

the proposition that computer assistance, through a DSS, can improve 
human decision performance: that is, that DSS-aided decision making 
is more effective and efficient than similar non-aided decision 
making. 


A laboratory experiment was conducted in which a management game 
was used to create a controlled decision environment. Six teams of 
Senior Executives were simultaneously exposed to two similar, non- 
structured decision making situations, one of which was directly 
aided by a decision support system. The research design permitted 
unobtrusive data collection for eight replications of the experi- : 
ment. For both DSS-aided and non-aided decision making, effective- 
ness was measured by the quality, consistency, and rate of improve- 
ment in decision making. Efficiency was measured by resource use 
and resolution time to final choice. 


Results significantly confirmed three hypotheses related to 
effectiveness (Hl, H2, H3) and two related to efficiency (H4, H5). 
The study showed that: 


For non-structured tasks, DSS-aided decision making 
resulted in decisions having: 

Hl: higher quality, 

H2: greater consistency, 

H3: higher rate of improvement, 

H4: lower resource use, and 

H5: shorter resolution time 
than non-aided decision making in similar, controlled 
circumstances. 


Thesis Supervisor: Michael S. Scott Morton 
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CHAPTER 1 


DECISION SYSTEMS IN PERSPECTIVE 


The potential of computer-based system for improving management 
decision performance has long been recognized. Pioneers in the 
field were speculating on the benefits of interactive systems in 
the early 1960's. Licklider (61) represents these early views: 


"The hope is that in not too many years, humans 
and machines will be coupled closely together 
and that the resulting partnership will think 
as no human has ever thought..." 


Gorry and Morton (49) suggest that early efforts were hindered by 
technological and conceptual barriers but, today, technology is 


no longer a problem. 


Nonetheless, present evidence suggests that interactive 
computer systems have had only limited impact on management 
decision making in the field. Brady's (17) recent study of 100 
top managers in major corporations concluded that computers have 
not had much of an effect on upper-level decision making. He 
found no evidence of direct use of computers by top management, 


but some influence was found at the middle management levels. 


In contrast, Scott Morton's (95) research shows that management 
decision systems have a significant impact on the decision making 
processes of higher level executives. He found decision makers 
adopt and use a computer-based system on a regular basis. The 
contact and elapsed time to problem rese@lution decreases, more 
problems are found, more alternatives are generated and evaluated, 
and communications between managers are substantially improved. 
Gerrity (45) and Hedburg (53) also report similar observations. 

The three studies supply strong evidence that the limited impact 
in the field may not be indicative of the true potential of these 


systems. 
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1.1 Decision Systems: Conceptual Distinctions 


Decision Systems are closely-coupled, man-computer partner- 
ships which focus on management decision making. The basic idea 
behind Decision Systems is the concept of comparative advantage. 
Whitfield (116) proposes that the fundamental notion is that men 
and computers have complementary talents. Emery (4) claims these 
systems must draw upon the best capabilities of both man and 
computer to deal with management problems that are too ill-defined 
and complex to be handled well by either partner alone. Miller (71) 
insists that the principal objective is to improve management 
decision performance; that is, to improve the effectiveness and 


efficiency of management decision making. 


Decision Systems involve a set of interacting components 
working together to make decisions and solve problems. The 
primary components of a Decision System are the 

(1) Decision Maker 

(2) Decision Support System 
The terminology adopted throughout this thesis is to refer to the 
total system as the Decision System (DS), to the human component 
as the Decision Maker (DM), and to the machine component as the 
Decision Support System (DSS). A Decision System is illustrated 


in Figure 1.1. 


Input Output 
Information Decisions 


Decision Maker 


(DSS) 


Decision Support System 


Figure 1.1 Decision System 
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The diagram illustrates important distinctions. First, 
Decision Systems have two components: a human component and a 
machine component. The machine supports the human in transforming 
information into decisions which result in achieving the goals 
of the decision maker. The extent to which goals are attained 
determines the effectiveness of the system. Effectiveness is a 
measure of external performance, the ability of the system's 
outputs — the decisions - to achieve desired ends. The expediency 
with which the system transforms information to decisions determines 
the efficiency of the system. Efficiency is a measure of internal 
performance, the ability of the system for transforming inputs 


into outputs. 


Despite the fact that performance is the result of both 
components working together, the performance of the system is 
commonly attributed to the human component. Consequently, the 
goal of improving system's performance is synonymous with improving 
the effectiveness and efficiency of the decision maker in the 


system. 


The decision maker can be either an individual, or more likely, 
a team working in concert with the machine. Thus, it might be more 
advantageous to regard the decision maker as the human component 
in the Decision System. This component is assumed to bring 
substantial skills to the Decision System, some of which may be 
difficult or impossible to provide in any other way. On the other 
hand, the human component may be subject to certain limitations 
which may be difficult to modify or overcome directly; that is, by 
changing its inherent capacity without aid from the Decision 


Support System. 


The Decision Support System is a group of programs working as 
a system to aid management decision making. The software of these 
sophisticated interactive systems provides this capability through 
decision aids, access to models, information, and computational 


power. This component is designed to support directly parts of the 
13 


management decision making process and, thus, provide assistance 
in the solution to complex, non-structured problems. To do this, 
the system must have some of the characteristics discussed in the 


next section. 
1.1.1 Characteristics 


Decision Systems are characterized by certain features which 
distinguish them from other computer-based systems, particularly 
Management information systems and data processing systems. 
Decision Systems interface with these systems through data and 
model bases. Ness (78) contends that data processing systems are 
the conventional transaction-oriented systems which support many 
operations in an organization. Blumenthal (13) asserts that 
management information systems supply information in standardized 
report form but usually lack the directness of support and other 
capabilities which characterize Decision Support Systems. 

Hedburg (53) stresses the need for adequate data collection systems 
which supply data for other systems. The relationships are 


pictured in Figure 1.2. 


Goals. _® decisions 


Decision Support 
System 


(DMB) 
Data/Model Bases 


(MIS) 
Management Information System 
External (DCS) Internal 
Information Data Collection System Information 
(DPS) 


Data Processing System 


Figure 1.2 Computer-based Systems 
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The purpose here is to briefly summarize the more important 
characteristics which distinguish Decision Support Systems from 
other computer-based systems. Gerrity (45), Hedburg (53), 

Ness (78) and Scott Morton (93) claim that Decision Systems are 
distinguished by their (1) capabilities and (2) directness of 


support. 


Decision Support Systems provide capabilities for interactive 
use of decision aids, information, models, and computational power. 
Scott Morton (95) contends that interactive models are necessities 
for Decision Systems. Gerrity (44) recognizes the need for 
memory (data bases), plans (models), and operators (decision aids). 
These capabilities support decision making processes by aiding 
in the retrieval, manipulation, and display of information for 
problem finding and in the identification, evaluation, and choice 
of alternatives for problem solving. Emery (39) asserts that 
these capabilities are necessary because the problems are non- 
structured and, thus, can not be solved effectively by either 


the man or the computer alone. 


Decision Support Systems directly support management decision 
making. Direct support refers to the close coupling of man and 
machine. Terminals provide the decision maker with on-line 
access to the Decision Support System. The decision maker and 
system usually communicate in a conversational manner; no inter- 
mediate programming is required. Results are returned directly 
to the user's terminal in a sufficiently short time; therefore 
the natural flow of the human decision process is not materially 
hindered. These systems do not require a structural change in 
the manager's role. He controls the decision making activity; 

a high premium is placed on his judgment and skill. Decision 
making is facilitated because the manager interacts with a 
Decision Support System, not because the decision making is built 


into the system itself. 
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1.1.2 Current Status of the Research 


This section clarifies the nature of this study by developing 
a framework for categorizing research. Presently, there is no 
adequate conceptual structure for classifying findings. The 
problem is recognized by Edstrom (36), Parsons (82), and 
Miller (71); each cite the need for an adequate taxonomy for 
structuring relevant research. The lack of an adequate framework 
has hidden the fact that the traditional research on decision 
making is, at best, only partially relevant for Decision Systems. 


Research on decision making is distinguished by: 

(1) focus of research 

(2) type of decision making 

(3) degree of task complexity 
These criteria are used in Figure 1.3 to suggest that research may 
focus on assessing the impact on decision processes and performance 
for both aided and non-aided decision making in situations which 
range from simple to complex. 


Focus of Research 
Decision Decision 
Processes Performance 


Task Decision Makin 
Complexity Esp 
“® Non-aided 
Simple 


Figure 1.3 Classification of Research 
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Decision Systems research is concerned with computer-aided 
Management decision making in complex problem situations. This 
is in sharp contrast with the traditional studies of decision 
making. Despite a notably large research effort most studies 
focus on non-aided decision making for simple task situations. 
For these reasons, the shaded parts of Figure 1.3 indicate that 
classical decision making research is only indirectly related to 
Decision Systems and, therefore, apparent relevance may be 


misleading. 


Decision Systems research is depicted in the non-shaded parts 
of Figure 1.3. As illustrated, it centers on assessing the 
effect of computers on decision processes and performance for 
complex, non-structured task situations. The work of Gerrity (44), 
Hedburg (53) and Scott Morton (95) falls into the upper left 
section. These field studies of computer-aided management decision 
making focused on assessing the impact on decision processes for 
complex task situations. These few studies are the most notable 


research reported in the literature. 


Unfortunately, there is no reported research which falls into 
the upper-right section. That is, there are no studies that focus 
specifically on decision performance for computer-aided management 
decision making involving complex, non-structured problems. The 
lack of research in this area stems from the inherent difficulty 
in measuring performance for complex situations. For studies that 
permit an objective evaluation of performance, most of the 
research has remained relatively inaccessible except to those 
directly involved in the research, especially for proprietary systems. 
These obstacles have hindered research in this area but have not 
lessened the need for studies on computer-aided decision performance. 

This thesis is an experimental study of computer-aided executive 
decision performance for complex, non-structured tasks and, thus, 


falls naturally into the upper-right section of Figure 1.3. 
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1.2 Motivation for this Study 


The motivation for this study was the need for knowledge 
concerning computer-aided decision performance. More information 
is required at this point. Even though there is voluminous 
literature on decision making and problem solving, little is 
known about computer-aided decision performance, especially for 


complex non-structured task situations. 


There is no dearth of assertions concerning the impact which 
Decision Systems have on management decision performance. These 
claims, however, are not based on research findings. Carroll (22) 
notes that the potential of Decision Systems to improve performance: 


"has not been rigorously demonstrated in 
the laboratory nor in the field." 


Despite this fact, Newman (81) contends that in the future more 
emphasis will be placed on improving higher level management 
decision performance through the use of interactive computer-based 


systems. 


Computer-based systems having the characteristics discussed 
in previous sections are called Decision Support Systems. Actual 
systems displaying some of these features are described in 
Austutz (4), Edstrom (36), Gerrity (44), Hedburg (53), Ness and 
Sprague (78) and Scott Morton (91). Scott Morton (91) notes that 
"the evidence collected thus far establishes beyond any doubt that 
such systems can be built with current technology". Similar 
conclusions have been reached by Boulden and Buffa (14) who claim 
"experience shows that a manager will eagerly use a computer in 


decision making if it is fast, economical and easy to work with". 


It is apparent that these systems can be built and may be used, 
but their impact on management decision performance is not clear. 
A basic question is: can Decision Support Systems improve management 
performance? This question was the principal motivation for this 


study. 
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1.2.1 Purpose of the Study 


Despite the tremendous interest in computer-aided decision 
making, there is a paucity of research evidence concerning management 
decision performance. Observations of prototype Decision Systems, 
however, suggest Decision Support Systems may improve management 
decision performance. In particular, the work of Gerrity (45), 
Hedburg (53) and Scott Morton (95) allude to the potential for 
improvement. Their observations suggested the basic thesis of 
this work: 

Decision Support Systems can improve management 
decision performance; DSS-aided decision making 


is more effective and efficient than similar 


non-aided decision making. 


The principal research objective is to test the validity of 
this assumption in an experimental study. The proposition is 
translated into three hypotheses related to effectiveness and two 
hypotheses related to efficiency. A management game is used to 
simulate a realistic decision environment which contains two 
comparative non-strutured decision situations. Decision making in 
one situation is aided by a Decision Support System (DSS) while 
decision making in the other situation is unaided. The gaming 
exercise provides data on actual results for DSS-aided and non- 
aided decision making. The quality of the decision making is 
measured for both conditions. This data is used for testing 
effectiveness hypotheses. Other data collected during the 


experiment provides information for testing efficiency hypotheses. 


The long range goal of this research is to contribute to a 
better theoretical base for Decision Systems by testing this basic 
tenet. The classical method of scientific inquiry - experimentation, 
observation and measurement - is used in this investigation of 


computer-aided executive decision performance. 
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1.2.2 Plan of the Study 


The contents of this study are divided into seven chapters. 
The content and organization of the main chapters are shown in 
Figure 1.4. The purpose of the figure is to illustrate the 
flow between sections and chapters and to briefly indicate the 
material covered by each chapter. Since each chapter provides 
a more detailed listing of its contents in the introduction, 


this section merely highlights their main parts. 


Chapter 2 discusses the nature of Decision Systems performance 
and shows how it relates to the concepts of effectiveness and 
efficiency. These concepts are operationally defined and the 
thesis is translated into hypotheses for the study. The require- 
ments for testing these hypotheses are used to develop a research 
design. An overview of the design is presented and the components 
of validity are discussed. This section serves as an introduction 
to the next three chapters since each details specific part of the 
total design and each discusses implications for the corresponding 


type of validity. 


Chapter 3 describes the design of the management game that 
simulates the decision environment in the experiment and the 
design of the decision aids that provide the computer-based 
decision support in the game. Implications for external validity 
that relate to the game and the decision aids are discussed in the 


last section of the chapter. 


Chapter 4 focuses on the design and conduct of the experiment. 
The structure of the design is explained as an introduction to the 
description of how the experiment was conducted. The experimental 
design's implications for validity are discussed and the quality 
of the design is judged in relation to specific criteria found in 


the literature. 


Chapter 5 details the procedure that was used to measure 


decision quality for both DSS-aided and non-aided decision making. 
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The procedure relates to the nature of the game and the design of 
the experiment. Issues concerning reliability of the procedure 


are discussed and its validity is evaluated. 


Chapter 6 presents the results of the study. Each hypothesis 
is formally tested and the results interpreted. The quality of 
the data for each hypothesis test is reviewed to determine the 
confidence that can be placed in results. Statistics on the 
use of decision aids are presented. The use of the Decision 
Support System is tied to the results of the hypotheses tests. 

The patterns of decision quality are compared to determine 


their similarities and differences. 
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CHAPTER 2 


DECISION SYSTEMS PERFORMANCE 


Decision Systems are man-computer partnerships whose main 
goal is to improve the effectiveness and efficiency of management 
decision making and, thus, improve management decision performance. 
Decision performance results from the interaction of the human 
and machine components in a Decision System. Therefore, decision 


performance must be considered from a system perspective. 


This chapter discusses the nature of decision systems 
performance, presents the hypotheses for this study and gives an 
overview of the research design. The organization and contents 
of the chapter are shown in Figure 2.1.. 

2.1 Decision Systems: Main Objectives 


2.1.1 Performance 
2.1.2 Effectiveness and Efficiency 


2.2 Decision Effectiveness Hypotheses 


2.2.1 Decision Quality 
2.2.2 Decision Consistency 
2.2.3 Decision Improvement 


2.3 Decision Efficiency Hypotheses 


2.3.1 Resource Use 
2.3.2 Resolution Time 


2.4 Research Design to Test Hypotheses 


2.4.1 Overview of the Design 
2.4.2 Validity in Design 


2.5 Chapter Summary 


Figure 2.1 Contents of Chapter 2. 


The first section discusses the nature of performance, effec- 
tiveness and efficiency. The second and third sections translate 
these concepts into hypotheses and reviews relevant research. The 
fourth section provides an overview of the research design and 
reviews the nature of validity; this serves as an introduction to 


chapters 2, 3 and 4. 
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2.1 Decision Systems: Main Objective 


The principal objective of coupling a manager and a computer 
in a Decision System is to improve management decision performance. 
Gerrity (45) insists: 


"the value of a man/machine Decision System 
should be measured by its ability to improve 
decision making performance." 


Newman (81) contends that decision performance relates to the 
effectiveness and efficiency of management decision making. 
Improving management decision performance, therefore, is 
synonymous with improving the effectiveness and efficiency of 
management decision making. These improvements result from the 


aid supplied by the Decision Support System. 


This perspective provides the basic thesis of this work: 


Decision Support Systems can improve human 

decision performance; DSS-aided decision making 

is more effective and efficient than similar 

non-aided decision making. 
This contention is important; it is a basic tenet in the field. 
Despite its significance, it is remarkable that it has never been 
fully tested. It is important, therefore, to regard this 
principle as an assumption since no systematic test of its 
validity has been made under controlled, laboratory conditions. 
The objective of this dissertation is to evaluate the validity of 
this contention. All of the hypotheses developed here relate to 


some aspect of this statement. 


To evaluate the validity of this compound thesis, each part 
is treated separately. It is entirely possible that DSS-aided 
decision making might be more effective but less efficient or 
the opposite might be true. In either case, the statement would 
be false. 
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Therefore, the statement is translated to: 


DSS-aided decision making is more effective 


than similar non-aided decision making. 
and 
DSS-aided decision making is more efficient 
than similar non-aided decision making. 
Hypotheses related to effectiveness are distinguished from those 
related to efficiency. This is also necessary because 


effectiveness and efficiency measure different aspects of systems 
performance. 


To measure performance requires adopting a systems perspec- 
tive. Gerrity (45) laments: 
"even now, Man-Machine Decision Systems are 
not yet widely viewed or studied as systems. 
Rather the computer and human components of 


the system still are often treated separately 
...to the detriment of the total system." 


Recognizing this, the systems perspective is adopted here. 
It is particularly important when considering the performance of 


the system shown in Figure 2.2. 


Input Output 
Information Decisions 


Decision Maker 


(DSS) 


Decision Support System 


Figure 2.2 Decision System 


This figure is the common reference for the following discussion on 
performance and its relation to measuring effectiveness and 
efficiency for a Decision System. 
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2.1.1 Performance 


Decision Systems performance is a combination of human and 
machine performance. Even though the dual source is unquestionable, 
Parsons (82) claims systems performance can not be separated into 
component performance because of the confounding interaction of 
each component. In other words, neither the man nor the machine 


accounts solely for the performance of the system. 


Despite the fact that performance is the result of man and 
machine working in concert, the performance of the system is 
commonly attributed to the decision maker rather than the Decision 
Support System. There are two reasons; first, the decision maker 
has the responsibility for the decision and second, the Decision 
Support System is controlled by the decision maker. Consequently, 
the decision maker ultimately determines the outputs - the decisions - 


of the system. 


What is needed is a method for measuring systems performance. 
Criteria for evaluating performance in Decision Systems experi- 
mentation must account for the joint performance of the decision 
maker and support system, not performance of one or the other 
alone. Hall (51) proposes that systems performance has two 
components: effectiveness and efficiency,and that the output of 
the system is the basis of measurement for effectiveness while 
resource use is the basis of measurement for efficiency. 
Effectiveness is a measure of the external performance of a 
system, Efficiency is a measure of internal performance. 

In short, by recognizing the differences between effectiveness 
and efficiency and using proper measures, the performance of 
the system,-as a whole, can be evaluated and not just one or 


the other of its components, 


26 


2.1.2 Effectiveness and Efficiency 


The concepts of effectiveness and efficiency are quite 
distinct. The efficiency of a system provides few clues about 
its effectiveness. Similarly, the effectiveness of a system may 
bear no relationship to costs of operation. Together, however, 
these concepts provide a means for measuring total systems 


performance. 


The concepts of effectiveness and efficiency can not be 
measured directly, however. They must be operationally defined. 
This requires translating each concept into specific constructs 
which can be reliably measured, and stating the conditions under 
which measurement will occur. The purpose of this section is 
to develop the concepts of effectiveness and efficiency and show 


how they will be measured for this study. 
Effectiveness 


A system is effective when desired objectives are obtained. 
Similar definitions are found throughout the literature by 
Ackoff (2), Boulding (15) and Hall (51). Decision Systems 
effectiveness, therefore, relates to the potential for reaching 
the objectives of the decision maker. Anthony (5) suggests: 

"Effectiveness relates to accomplishment... 
when a specific desired end is attained we 
shall say that an action is effective." 

The concept of effectiveness has not been operationalized 
for Decision Systems. Nevertheless, it is clear that effectiveness 
is a measure of how well the outputs of a Decision System attain 
the goals of the decision maker. Measured directly in terms of 
the system's outputs, effectiveness shown in Figure 1.2 is 
determined by the: 

(1) quality of the decisions 
(2) consistency of the decisions 


(3) improvement in the decisions 
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The primary goal of Decision Systems is to increase profits by 
improving decision effectiveness through better quality, consistency 


and rate of improvement in decision making. 


Efficiency 


A system is efficient when it transforms input to outputs 
in a reasonably expeditious manner. Anthony (5) states that 
efficiency means the ability to produce greater outputs with 
the same inputs or the same outputs with fewer inputs. Boulding (15) 
claims the most common measure of efficiency is the ratio of 
outputs to inputs but this ratio cannot be used as a measure of 
efficiency when the input and output which pass through the 
systems boundary remains constant. For this case, Hall (51) 
proposes that the measure of efficiency must be based on resource 


use and associated measure related to the transformation process. 


These observations suggest two dimensions related to 
efficiency of the system: 
(1) resource use 
(2) resolution time 
Another major goal of Decision Systems is to improve system 
efficiency by reducing the resource use and the elapsed time to 


decision resolution. 


Summary 


The nature of decision performance is summarized in Figure 2.3. 
Decision performance is composed of twin components - effectiveness 
and efficiency - which, in turn, have a number of dimensions. 
Effectiveness relates to the quality, consistency and rate of 
improvement in decision making. Efficiency relates to resource 
use and the resolution time. These relationships are shown 


in the figure on the following page. 
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Decision Performance 


Decision Decision Decision 
Making Quality Consistency 


(H1) 


| Computer 
| Aided 


| Non-Aided 


The basic thesis can now be translated to specific hypotheses 


Figure 2.3 Decision Performance 


for effectiveness and efficiency. All hypotheses will be subject 
to the same experimental conditions which have two salient 
characteristics. First, hypotheses will be tested in situations 
where decision making involves complex, non-structured managerial 
tasks. Second, the experimental design will permit comparison of 
DSS-aided and non~-aided decision making under similar, controlled 
circumstances. Therefore, each hypothesis statements will compare 


these two conditions. 
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2.2 Decision Effectiveness Hypotheses 


Hypotheses related to decision effectiveness are introduced 
briefly here. Details and relevant research are presented in 


separate sections. The effectiveness hypothesis is: 


For non-structured managerial tasks, DSS-aided 
decision making is more effective than 


non-aided decision making in similar controlled, 
circumstances. 


The preceding discussion has shown that effectiveness is a 
multidimensional concept. Because of this, the statement will 
not be tested directly. Instead, hypotheses related to each 
dimension of effectiveness will be tested. This statement 


translates to three supporting hypotheses which are: 


For non-structured managerial tasks, DSS-aided 
decision making will result in decisions whose: 


(H1) - quality is higher 
(H2) - quality is more consistent 
(H3) - quality has a higher rate 


of improvement 


than non-aided decision making in similar, 
controlled circumstances. 


Each of these hypotheses relates to the level or pattern of 
decision quality. To better illustrate what is expected, details 
of each hypotheses are discussed in reference to the same 
theoretical graph shown in Figure 2.4. The graph illustrates 
theoretical levels and patterns of decision quality over time 
which might be expected for DSS-aided and non-aided decision 


making. 
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Figure 2.4 Theoretical Decision Quality 


Both DSS-aided and non-aided quality levels are shown to 
emphasize the comparative nature of this study. The graph is 
predicated on the belief that decision makers can adopt and use 
a Decision System for non-structured problem solving. In the few 
instances in which these systems have been built, decision makers 
have made meaningful use of fairly sophisticated systems; Ferguson 
and Jones (42), Gerrity (45), Hedburg (53), Morton (92). These 
studies have also shown that decision makers quickly learned to 


exploit the capabilities of the particular Decision System. 


This graph is not exact but is merely used to illustrate 
expected levels, variations and trends for DSS-aided and non-aided 
decision quality over time. Since decisions are made at discrete 
points in time, the quality of decision making is a series of 
points rather than a continuous function. For purposes of 


exposition, however, a broken line is used to clarify expectations. 


The graph represents decision quality expected from groups 
working on similar problems. Howard and Morgenroth (54) found 
similar decision environments produce essentially the same 
decision processes and patterns of decision performance. Moskowitz (74) 
found that groups experience variations in decision quality as 
task complexity changes. Therefore, variations in quality are 


expected for DSS-aided and non-aided decision making. 
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2.2.1 Decision Quality 


Hl: For non-structured managerial tasks, DSS-aided decision 
making will result in decisions whose quality is higher than non- 
aided decision making in similar, controlled circumstances. 


DSS-saided 


Decision 
Making 
Quality 


~Non-aided 
a” 


Time 
Figure 2.5 Decision Quality Hypothesis 


Figure 2.5 graphically illustrates the decision quality 
hypothesis. DSS-aided decision making quality is expected to be 
better than non-aided decision quality for similar circumstances. 
Comparative quality levels are expected to vary over time but 
DSS~aided decision quality is expected to remain higher than 
Similar non-aided decision quality. The difference in quality 
levels is expected to vary with the degree of aid provided by 
the Decision Support System. Similar fluctuation in quality are 
expected to occur at the same points in time but decreases in 
DSS-aided quality are expected to be smaller than decreases in 
non-aided decision quality. In summary, DSS-aided decision making 
will result in decisions whose quality is higher than the quality 


of non-aided decision making for similar situations. 
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Relevant Literature 


For comparative situations, DSS~aided decision making quality 
is expected to be better because Decision Systems may decrease 
inherent human limitations and, therefore, aid the decision 
processes of managers. Beged-Dov (11) proposes that Decision 
Systems must supply relevant information, models and aids for 
manipulating, recording and transforming data. Simon (103 ) 
claims that this allows the decision maker to allocate his limited 
decision making resources and capabilities on fewer tasks within 
the total decision process and in so doing may improve his 
decision quality. Newman ( 81 ) claims that if we concentrate on 
providing computer and display aids that overcome limitations, 
then decision making abilities can be expanded tremendously. These 


contentions are discussed below. 


Decision Systems facilitate decision making. These systems 
can aid all phases of the decision process and may increase the 
decision maker's ability to find problems, evaluate consequences, 
and make choices. Gerrity ( 44 ) reports that DSS-aided decision 
makers found problems that may have gone unrecognized. Morton ( 95 ) 
notes that more alternatives were generated and tested in less 
time with the aid of his Decision Support Systems. It seems 
clear that decision processes have been significantly influenced 
by Decision Systems. The implication is that decision quality may 


also have been changed but this has never been substantiated. 


Decision making is shaped by the capabilities and limitations 
of the decision maker. Howard and Morgenroth ( 54 ) modeled the 
decision processes of 130 non-aided decision makers and concluded 
that, "decision processes of higher level executives displayed a 
surprising degree of simplicity, a simplicity imposed by the 
limitations of man's intellectual capacities." 

The capabilities of the human decision maker are quite limited. 
Schakle ( 98 ) found that humans are limited in computational 
power and precision. Newman ( 80 ) reported limited ability to 
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handle complexity. Miller (68 ) asserts that short term memory 
tends to be limited and unreliable. Hunt and Markos ( 57 ) 
reported limited information handling ability. Bruner ( 18 ) 
show that the decision maker is severely constrained by his 
inference capabilities. Dickson ( 33 ) found that the ability to 
process many facts at the same time is poor, especially for 
unfamiliar tasks. The exact boundaries of these limitations are 
not clear. Newman ( 81) asserts that, "there are, undoubtedly 
upper limits to man's intellectual ability, but we are a long 


way from determining just where those limits are." 


These limitations contribute to illogical and suboptimal 
decision making in complex situations where few aids are provided. 
Ebert (35 ) reported computer aids overcome some of these 
characteristics, especially for complex problems. There is additional 
evidence that performance can be improved. In one of the most 
extensive studies on computer-aided problem solving, Newman and 
Rogers ( 80 ) reported slight differences in performance between 
control and experimental groups for simple tasks requiring concept 
formation and inductive reasoning. In military command situations, 
Gebhard ( 43 ) found computer-based solutions to simple problems 
were used as a basis for decision making and led to improved decision 


quality. 


The strongest indication is found in the work of Gerrity (44 ) 
and Morton ( 95 ). Although both field studies focused on the impact 
on decision processes rather than performance, each alludes to 
increased decision quality. Morton ( 95 ) documented significant 
impacts on the decision process but observed: 


"In point of fact, there is no certain way of showing 
in general that decisions are "better'...(but)... 
these particular managers claimed that MDS improved 
decision making as well as shortened the decision 
making cycle..." 


Gerrity ( 44) and Hedburg(53 ) have reported similar observations. 
Taken together, the findings and observations suggest DSS—aided 
decision quality can be higher than non-aided decision quality for 


situations of similar complexity. 
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2.2.2 Decision Consistency 


H2: For non-structured managerial tasks, DSS~aided decision 


making will result in decisions whose quality is more consistent 
than non-aided decision making in similar, controlled circumstances. 


Decision 
Making 


Quality 
aNon-aided 


Time 


Figure 2.6 Decision Consistency Hypothesis 


The consistency of DSS~aided decision making is expected to 
be better than the consistency for non-aided decision making. 
Consistency is a measure of the variance in decision behavior. 
As shown in Figure 2.6 variation within the dotted range lines 
is expected to be smaller for DSS-aided decision making than for 
non-aided decision making. Quality levels are expected to vary 
as the complexity of the decision environment changes but DSS- 
aided quality is expected to show smaller comparative decreases 
for similar circumstances. The smaller decreases are in sharp 
contrast to the major decreases expected for non-aided decision 
making. In summary, DSS-aided decision making will result in 
decisions whose quality is more consistent than the quality of 


non-aided decision making for comparative circumstances. 
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Related Literature 


For similar situations, DSS-aided decision making is 
expected to be more consistent because Decision Systems can 
provide formal models tailored to the need of the decision maker. 
Models provide a consistent structure of relationships and thus, 
produce logically consistent results. Morton (97 ) contends 
that the direct availability of models is the feature that 
distinguishes Decision Systems from the more traditional 


Management Information System. 


Research suggests that decision performance may be improved 
by providing models and, thus, making decision rules more 
consistent over time. Bowman's ( 16 ) research shows that 
managers make good decisions on the average but also may exhibit 
high variance in their decision behavior. He asserts that models 
based on parameters determined from the manager's actual past 
decisions and centering on critical variables are the key to 
more consistent decision making. Similar arguments for model 
based decision making are proposed by Little (63), Gorry and 


Morton ( 49 )and Charnes and Cooper ( 24 ). 


Kunreuther ( 60) claims that managers consider only a 
limited number of factors in their decision making. The 
implication is that decision making is likely to be geared to 
a decision rule based on a few variables which the manager has 
found to be important from past experience. Therefore, it seems 
that the non-aided manager may be erratic in his decision making 
behavior since he considers only the variables that he is able 
to handle using his limited models. Thus, variability may stem 


from a lack of good models. 


Non-aided decision makers exhibit considerable inconsistency 
in their information processing behavior. Schroder ( 96) asserts 


that the capacity for processing information tends to increase, 
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passes through a maximum and falls off with increasing complexity 
but that the tolerance for complexity increases with experience. 
Newman ( 81) reported decision makers are better able to handle 
complexity if they have some way of structuring it. Nevertheless, 
these findings suggest that decision consistency can be increased 


simply by structuring complexity with a model. 


It is likely that model-based decision making introduces 
greater consistency in real-world decision process. 
Westendorf (114) showed that managers in an experiment group 
supported by models improved both long and short range decision 
consistency over the control group in a complex game. Additional 
research verified that the relative superiority of model-based 
decision making improves as task complexity increases. 
Morris (73) reported confirming results which showed that models 
tend to increase consistency of decision making in actual 
situations. These results indicate that models reduce the chance 
of overlooking critical variables and may provide the ability 


to handle greater complexity. 


There are claims in the man-machine systems literature 
that decision consistency increases when decision making is 
aided by computer support. The evidence cited above suggests 
that DSS-aided decision making can be made more consistent 
than similar non-aided decision making by providing systems 
which supply information models and computational power to 


the decision maker. 
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2.2.3 Decision Improvement 


H3: For non-structured managerial tasks, DSS-aided decision 
making will result in decisions whose quality has a higher rate of 


improvement over time than non-aided decision making in similar, 
controlled circumstances. 


Decision 
Making 
Quality 


Time 


Figure 2.7 Decision Improvement Hypothesis 


The rate of improvement for DSS-aided decision making is 
expected to be greater than the corresponding rate for non-aided 
decision making. Rate of improvement is a measure of the net 
rate of change in decision quality over time. As shown in 
Figure 2.7 the slope of the line for DSS-aided decision quality 
is expected to be greater than the slope for non-aided decision 
quality. The quality of decision making in both states is 
expected to increase but DSS-aided decision quality is expected 
to approach potential faster. Therefore DSS-aided decision 
quality is expected to have a higher rate of improvement than 


non-aided decision quality. 
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Related Research 


For comparative situations, DSS-aided decision quality 
is expected to have a higher rate of improvement because 
Decision Systems can materially affect the decision maker's 
abilities. Since part of the total decision problem is shared 
by the machine, the decision maker can concentrate his resources 
on fewer tasks for which he has a comparative advantage. The 
net result should be that decision quality improves at a faster 


rate. 


The non-aided decision maker, on the other hand, is 
faced with formidable problems in non-structured situations. 
Since task complexity is determined by the nature of the 
environment, he has little chance of reducing it. Moreover, 
inherent human limitations of the non-aided decision maker 
can not be changed directly (e.g. direct increase in short- 
term memory). These two facts suggest that the rate of 
improvement for non-aided decision maker slowly increases as 
the decision maker learns to cope more by structuring parts 


of the total problem. 


These observations are supported by research related to 
human information processing capabilities. Powers ( 86 ) 
reported subjects came close to good solutions for very simple 
tasks but only in the latter stages of learning. A tentative 
finding was that poorer decision making in the initial and 
intermediate stages might be accounted for in part by the 
subject's lack of skill in considering simultaneously all 
relevant aspects of the task and their ability to process 


information. 
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Research concerning non-aided human information processing 
ability has implications for decision improvement. Gibson 
and Nicol (47 ) claim decision makers tend to want too much 
rather than too little information. McKendry (66 ) found 
Managers can judge the potential value of new information but 
are unable to effectively utilize relevant information, 
especially if it is multi-dimensional. Hunt ( 55 ) and 
Schroder ( 96 ) reported findings which suggest that there 
are many situations where better information may not be used 
simply because humans have difficulty aggregating available 
evidence. Consequently, non-aided decision makers find it 
difficult to improve or revise strategies even when warranted 
by new information. Vaughn (113 ) showed that limited 
information processing capacity results in decision makers 


generating and testing too few courses of action. 


These findings contrast with results reported for 
computer-aided decision making. Hedburg ( 53 ) found decision 
makers could handle more information. Morton ( 95 ) reported 
that decision makers generate and test more alternatives in 
the solution of complex problems. -Newman and Rogers ( 80 ) 
claim that fewer errors are made in arriving at a solution 
to concept formation problems. Newman ( 81 ) contends that 
decision makers increase their ability to handle complexity 
if the system structures information properly. This limited 
evidence suggests that the rate of improvement for DSS-aided 
decision making may be greater than for similar non-aided 


decision making. 
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2.3 Decision Efficiency Hypotheses 
Hypotheses related to the efficiency of decision making are 
introduced here. Further detail is provided on each hypothesis 


and research related to both is discussed in the following section. 


The efficiency hypothesis is: 


For non-structured managerial tasks, DSS-aided 


decision making is more efficient than non- 
aided decision making in similar, controlled 


earn perenne eee niente een nears A eS 
circumstances. 
The measures of efficiency are based on resource use and 
resolution time. Therefore, this major hypothesis translates 


into two supporting hypotheses which are: 


For non-structured tasks, DSS-aided decision 
making will result in: 


(H4) - lower resource use 


(H5) - shorter resolution time 


than non-aided decision making in similar, 
controlled circumstances. 


Even though Decision Systems include human and machine 
resources, only the human resource can be considered in evaluating 
these hypotheses because of the need for comparative measures of 
DSS-aided and non-aided resource use. Consequently, the measures 
used here are the number of man-hours devoted to DSS~-aided and 


non-aided decision making. 


These hypotheses relate to different aspects of decision time - 
contact time and elapsed time. Man-hours of effort is a measure 
of the contact time that decision makers use in arriving at decisions. 
Resolution time is a measure of the elapsed time from the beginning 
of the decision process to the final choice of an alternative. 


Together, they provide good measures of decision making efficiency. 
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2.3.1 Resource Use 


H4: For non-structured managerial tasks, DSS-aided decision 
making will result in lower resource use than non-aided decision 
making in similar, controlled circumstances. 


Man-Hours 


™ “Non-aided 


DSS-aided 


Time 


Figure 2.8 Resource Use Hypothesis 


Resource use is measured by man-hours of effort required for 
comparative decision activity. Fewer man-hours for DSS-aided 
decision making are expected to be used. Initially, man-hours 
are expected to be high in both cases because the firms: must make 
complex decisions in a new environment. Moreover, for aided 
decision making, they must learn to use the decision support system. 
As shown in Figure 2.8, after a few quarters, the man-hours for 
aided decision making are expected to decrease at a faster rate 


and stay below those required for non-aided decision making. 
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2.3.2 Resolution Time 


H5: For non-structured managerial tasks, DSS-aided decision 
making will result in shorter resolution time than non-aided 
decision making in similar, controlled circumstances. 


Resolutio 
Time 


ee 
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DSS-aided 


Time 


Figure 2.9 Resolution Time 


The resolution time - the time until final choice - is 
expected to be shorter in most cases for DSS-aided decision 
making. Resolution time can be measured objectively by recording 
the elapsed time from the beginning of decision making to choice 
of an alternative. As shown in Figure 2.9, resolution time of 
DSS~aided decision making is expected to lower and may decrease 
over time while resolution time for non-aided decision making is 


expected to be higher for similar circumstances. 
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Related Research 


Fewer man-hours of effort are expected to be used for DSS-~aided 
decision making because Decision Systems facilitate problem solving 
by providing immediate access to data, models and decision aids. 
Consequently, decision processes can proceed with minimal inter- 
ruption at a pace dictated by the decision maker. The flow of 
non-aided decision making, on the other hand, is usually subjected 
to artificial interruptions because of the need for time-consuming 
analyses by either the decision maker or his staff. Decision 
Systems can reduce artificial barriers to decision making and, 
therefore, reduce the man-hours of effort need to make required 


decisions. 


There is some evidence to support these contentions. Joyner 
and Tunstall (58) reported an increase in efficiency for computer- 
aided problem solving for fairly simple task situations. They 
claim that the increase in efficiency stems from reducing the 
barriers to problem solving and segmentation of the decision 
process into a series of distinct, logically sequenced steps. 
Moreover, this was followed by a reduction in the complexity of 


the information processing required at any given time. 


The most striking evidence is found in the work of 
Scott Morton (95). He reported that the contact time for three 
managers involved in a complex, non-structured marketing problem 
was reduced from the original six days to one-half day after the 
Decision System was introduced. The manager's contact time is 
different than elapsed time to decision. Elapsed time for non- 
aided decision making originally spread over twenty-two days. 
Sixteen days were used by the manager's staff to carry out detailed 
analyses which were reviewed by the managers as part of their 
decision process. Evidently, the quality of analyses was improved 
and the flow of the decision process was facilitated, thus reducing 
elapsed time to one day. Morton (95) attributes some of the 


reduction to a substantial improvement in communication among managers. 
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2.4 Research Design to Test Hypotheses 


To test the hypotheses discussed in previous sections of this 
chapter requires a suitable research design. Kerlinger (59 ) 
claims that the suitability of a particular design must be judged 
in terms of its appropriativeness for testing hypotheses and its 
inherent validity. Since alternative designs have differing 
degrees of inherent validity, the preferred design is one having 
the highest validity because validity determines the value of 


the work and the degree of confidence associated with results. 


Validity is a multifaceted concept whose nature is best 
understood by distinguishing among its basic components. 
Campbell and Stanley (20 ) identify three major components of 
validity: validity of results (external validity), validity of 
methods (internal validity) and validity of analytical procedures 
(procedural validity). Obviously, each component relates to 


specific parts of a research design. 


The design used in this study is quite complex. Three 
chapters (3, 4 and 5) are devoted to detailing specific parts 
of the total design and to evaluating their implications for the 
corresponding type of validity. This section serves as the 
introduction to these three chapters. The purpose here is: 


(1) First, to introduce the complete design and 
show how details in the following chapters 
fit together to produce data to test the 
hypotheses. 


(2) Second, to review the nature of external, 
internal and procedural validity and 
establish the correspondence between 
specific parts of the design and each 
component of validity. 


This organization parallels the organization of chapters 3, 4 
and 5. The first two sections of each chapter describe details of 
the design while the third section discusses implication for 
validity. The details of the design are provided first because 
they are necessary for evaluating critical issues related to each 
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2.4.1 Overview of the Research Design 

Research design is more than the design of an experiment. 
It includes all of the steps which eventually lead to hypotheses 
testing. The objective of any design is to produce appropriate 
data for the hypotheses tests. Different types of data are 
needed for the effectiveness and efficiency hypotheses. 
Consequently, these needs determine the required parts of the 
design. These needs are discussed below to show how they 


influenced the design. 


The data needed to test each of the effectiveness hypotheses 
are measures of actual decision quality for DSS-aided and 
non-aided decision making. A number of requirements must be 
met in order to obtain the data. A decision environment must 
provide two similar decision making situations. To create the 
experimental and control conditions, decision making in one 
environment must be aided by a Decision Support System while 
its counterpart in the other environment remains unaided. 

The actual results of decision making can be obtained directly 


by conducting the experiment. 


Extensive analysis is required to measure the quality of 
decision making. The quality of actual decision making can 
be measured by finding ratios of actual profits to potential 
profits which would have resulted had decision making been 
optimal or near optimal. These ratios are the data needed 


for testing hypotheses related to effectiveness. 


To test each of the efficiency hypotheses requires 
_ data on resources use and resolution time for DSS-aided 
and non-aided decision making. The data on man-hours 


used and the time to final choice can be collected during 
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the course of the experiment and used directly to test the 
hypotheses. 
Translating these needs into an appropriate design involved: 
(1) Design of the Simulation and Decision Aids 
(2) Design and Conduct of the Experiment 
(3) Design of Measures and Methods of Analysis 
The distinction between components of validity strongly 
influenced each of these areas. Early in the study, goals for 
increasing the validity of the design were developed. These goals 
are: 


Comparison of Aided and Non-aided decision making 
Adequate control of experimental variables 
Simultaneous exposre to experimental condition 
Parallel observations of conditions 

Unobtrusive data collection 

Replication of the experiment 

Reliable methods of analysis 

Realistic and complex decision environment 
Conditions having high generalizability 

10. Range of non-structured decision making 


e e e 
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These goals were achieved in the design pictured in Figure 2.10. 
Features of the design are described below. The purpose is to 
provide an overview of the design rather than specific details 


which are described in later chapters. 


A management game is used to simulate a complex business 
environment in which firms compete in a Foreign Market and 
Domestic Market. These markets are highly similar; they are 
structurally identical and are effected by exactly the same 
variables. The similarity, however, is not apparent to firms 
because of complex interactions. Six firms compete in both 
markets by making non-structured marketing decisions (price, 
promotion and R&D) which determines sales and, in turn, marketing 


profits. 


An integral part of the game is a set of interactive decision 
aids, one of which is a Decision Support System. This marketing 
Decision System is designed to directly support managers of each 
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firm. The Decision Support Systems can be supplied or withheld 


from either market to create experimental conditions. 


The experimental design creates the experimental and control 
conditions by making the Decision Support System available for 
use in the Foreign Market while precluding use in the Domestic 
Market. All other conditions are the same for all firms in both 
markets. Consequently, the experimental condition involves 
DSS-aided decision making in the Foreign Market while the control 


condition involves non-aided decision making in the Domestic Market. 


The design uses firms as their own control; that is, each 
firm is simultaneously exposed to the experimental and control 
conditions. Firms may be used as their own controls because 
results in the Foreign Market are independent of results in the 
Domestic Market. Therefore, differences in actual results are 
attributable to the experimental variable - the Decision Support 
System. 


Conducting the experiment is synonomous with allowing firms 
to play the game for enough quarters to supply adequate data. 
The actual results for each firm are collected for both the 
experimental and control conditions for eight quarters. A variety 
of unobtrusive data collection methods are used to obtain additional 
information. Among these methods are console traces, questionnaires 
interviews, observation and document collection. Most of this 


data is collected after the end of the gaming exercise. 


After the game has ended, decision quality for each firm's 
DSS-aided and non-aided decision making is measured for every 
quarter. A ratio of actual marketing profitability to potential 
marketing profitability which would have resulted had the firms' 
marketing decision been optimal or near optimal is used to measure 
relative quality. These measurements are the data for hypotheses 
tests related to decision effectiveness. In contrast, the data 
related to efficiency require no elaborate analyses prior to 
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2.4.2 Validity in the Research Design 


To conduct research which does not violate the canons of 
scientific procedure while producing results which are generalizable 
to a wider context requires that the design have high validity. 
Validity is much more a matter of degree than an absolute. Because 
every aspect of validity is so closely interwoven, at times, these 
criteria may be at odds with each other. Research designs which 
increase one may jeopardize the others. Clearly, the goal is a 
design that is high in external, internal and procedural validity. 
Every part of the research design has implications for validity. 

The correspondence between parts of the design and each type of 


validity should be evident from a discussion of each component. 


External Validity 


External validity centers on generalizability. It depends 
upon the similarity of the experimental and real world situations. 
Greater external validity is achieved as representativeness of 
the laboratory environment increases. The validity of results, 
however, is dependent on the validity of methods and procedures. 
In other words, results are valid only to the extent that methods 
and procedures are valid. Campbell and Stanley ( 20) note that 
external validity asks the question of generalizability: to what 
populations, settings, treatment variables, and measurement 
variables can effects be generalized? Raser (87) identifies 
the representativeness of the experimental (1) environment, 

(2) conditions, and (3) population as the critical issues for 


consideration. 


Certain factors related to these variables may jeopardize 
external validity and thereby decrease the ability to transfer 
results to a wider context. Of particular importance is the 
realism and complexity of the simulated environment and the 
similarity of the conditions and subjects in the Game to their 
real world counterparts. These factors relate to the design of 
the simulator, interactive decision aids and the participants in 


the MIT Management Game. 
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Internal Validity 


Internal validity is a measure of the quality of experimen- 
tation. It depends upon the quality of the design and procedures 
used in the experiment. Greater external validity is achieved by 
design which innately control variables that might confound the 
effects of the experimental stimuli. Parsons (82 ) suggests that 
for man/machine experimentation the factors influencing internal 
validity relate either to how the experiment was designed or to 
the procedures used in conducting the study. These factors are 
usually situation-specific and must be evaluated in the context 


of the particular experiment. 


The problems encountered in man/machine studies may be more 
severe than those in conventional experimentation because of the 
scope and complexity of the experimentation. Therefore, the 
researcher must explore the implications of all experimental 
factors which have the potential for jeopardizing either internal 
or external validity. Among the more important factors in this 
study are procedures used for the formation of experimental and 
control conditions, the participant assignment methods, the data 
collection procedures and methods of observation. These factors 


are associated with the design and conduct of the experiment. 


Procedural Validity 


Procedural validity is related to the quality of the measures, 
analytical procedures, and statistical tests used in the research 
design. It depends upon the quality of the methods for trans- 
forming raw data into appropriate hypotheses test. Mackenzie 
and Barron ( 64) note that there are many transformations which 
are required to obtain data for hypotheses testing and finally 
apply statistical procedures to obtain results and each has the 


potential for influencing procedural validity. 


Potential problems are the appropriativeness of measure§é, 


the reconciliation of differences for DSS-aided and non-aided 
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decision making, the assumptions behind analytical procedures, 
and the use of statistical tests. These factors relate to the 


design of measures and methods of analysis. 


Summary 


Validity is intimately related to each part of the total 
research design. External validity depends upon the similarity 
of this experimental and real world situation. Internal validity 
depends upon the quality of the experimental design and methods 
for conducting the experiment. Procedural validity depends upon 


the reliability of measures and methods of analysis. 


The previous discussion has shown that factors influencing 
each type of validity logically correspond to parts of the 


research design. The correspondence is: 


Type of Validity Research Design Chapter 
External Design of the Simulation and Decision Aids 3 
Internal Design and Conduct of the Experiment 4 
Procedural Design of Measures and Methods of Analysis 5 


The following three chapters first present details of the 
research design and then discuss implications for the 


corresponding type of validity. 


Each chapter uses data from the questionnaire and interviews 
to show participant response to the critical issues related to 
validity. Each section presents data in histogram form. Since 
each question which elicited the responses is printed with the 
histogram no further detail regarding this data is given until 


data collection methods are discussed in Chapter 4. 
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2.5 Chapter Summary 


This chapter documents underlying concepts, develops hypotheses 

and introduces the research design used in this study, The thesis that 
Decision Support System can improve human decision performance: that is, 
DSS - aided decision making is more effective and efficiency than similar 
non-aided decision making is translated into three testable hypotheses 
related to effectiveness and too related to efficiency. 
Effectiveness hypotheses are: 

For non-structured managerial tasks, DSS - aided decision 

making will result in decisions whose: 


(H1) quality is higher 
(H2) quality is more consistent 
(H3) quality has a higher rate of improvement 


Efficiency hypotheses are: 
For non-structured tasks, DSS - aided decision making 
will result in: 


(H4) lower resource use 
(H5) shorter resolution time 


than non-aided decision making in similar, controlled circumstances, 

Research related to each of these hypotheses is reviewed. The review 
reveals that these hypotheses have never been tested (even singularly) in 
a controlled laboratory experiment, 

The chapter also introduces the research design that was used to simul- 
taneously test each of the hypotheses. The goals of good research: high 
external, internal and procedural validity; are discussed and related to 
specific characteristics of the research design as an introduction to 
Chapters 3, 4, and 5 which explores details of the complex design used 


in this study. 
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CHAPTER 3 
DESIGN OF GAME AND DECISION AIDS 


The MIT Management Game exercise is a unique laboratory for 
the study of Decision Systems performance. Certain features of 
the Game create the opportunity to study computer-aided and non- 
aided decision making in a realistic decision environment. 
Interactive decision aids are provided to support decision making: 
Marketing Decision System (MDS), Financial Planning System FPS), 
and Statistical Analyses System (SAS). 


This chapter describes the Game and the decision aids and 
discusses their implications for external validity. The 
organization and content of the chapter is shown in Figure 3.1. 

3.1 Design of the Game 


3.1.1 Structure of the Game 
3.1.2 Nature of Decision Making 
3.1.3 Participants in the Game 


3.2 Interactive Decision Aids 


3.2.1 Marketing Decision System 
3.2.2 Financial Planning System 
3.2.3 Statistical Analysis System 


3.3 Implications for External Validity 


3.3.1 Issues Related to Game Environment 
3.3.2 Issues Related to Game Conditions 
3.3.3 Issues Related to Game Participants 


3.4 Chapter Summary 
Figure 3.1 Contents of Chapter 3 


The first section describes the details of the Game and the 
complexity of the marketing decision making facing the participants. 
The second section describes each of the decision aids. MDS is 
described in detail and an example of how firms used this system 
is given, Although the first two sections are brief, they provide 
the necessary detail for evaluating specific issues related to the 


external validity of this study. 
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3.1 Design of the Game 


From the firm's point of view, the Game is an exercise in 
executive decision making. Participants act as upper-level 
management teams competing in a Foreign Market and Domestic Market. 
Management teams control their firms by making marketing, production 
and financial decisions which determine the financial and operating 


results that are reported at the end of each period. 


Each firm is required to make sixteen decisions every quarter. 
The decisions fall naturally into three interdependent functional 


areas. The required decisions associated with each area are: 


Function Area Required Decision 
Marketing . Domestic Price 


1 
2. Foreign Price 

3. Domestic Promotion Expenditure 
4, Foreign Promotion Expenditure 
5. R&D Expenditure 


Production 6. Units to be Produced 
7. Shipments to Foreign Market 
8. Machine Capacity 
9. Labor Force 

Pinance 10. Cash Remitted 


11. Domestic Securities Balance 
12. Foreign Securities Balance 
13. Domestic Loan Balance 

14. Foreign Loan Balance 

15. Number of Shares Outstanding 
16. Dividend 


The results returned to each firm include both public and 
private information. All firms receive confidential reports 
concerning their financial and operating status which includes 
over 250 information items organized into ten reports: 

Balance Sheets 

Profit and Loss Statements 

Reconciliation of Retained Earnings Statements 
Sources and Uses of Funds Statement 

Inventory Reconciliation Statement 


These ten reports are included in Appendices C and D. 
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Since all companies are considered publicly held, each firm 
also receives an identical "market summary" report which contains 
information on each market's data and competitive data that is 
normally available in the market place. A copy of this report is 


included in Appendix B. 


To better illustrate the sequence of events, Figure 3.1 high- 
lights the coneptual operation of the game. In essence, it is a 
"behind the scenes" look at what happens when firms submit their 
decisions on the Data Input Form. A copy of this form is shown 
in Appendix A. This form also captures data for experimental 
purposes. After the decisions for all firms are entered, the game 
is run by the administrator and results are returned. The input 
required from each firm and output returned as well as the 


conceptual activities in the game are pictured in the diagram. 
3.1.1 Structure of the Game 


The purpose of this section is to describe the Game because 
the generalizability of results depends upon the realism and 
complexity of the experimental environment. The realism and 
complexity of the Game is related to the nature of the (1) markets, 


(2) customs, (3) industry and (4) firms. 


The Game simulates a multi-firm industry serving two indepen- 
dent markets - the Foreign Market and the Domestic Market. Each 
market is independent in that sales in one do not influence sales 
in the other. Six firms have by tradition been considered an 
industry. Since activities of the industry represents the strategies 
of all firms, the dynamic interaction of many variables influence 


the growth or decline of each market. 
Markets 


Markets react to general economic conditions and the industry's 
price, promotion, and research and development policies. Changes 


in each market's size are determined by general economic conditions 
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and the actions of each firm in the industry. Changes in GNP 

and a seasonal factor which represents a quarterly sale pattern 
influence growth. The interaction of each firm's price, promotion, 
R&D, and inventory policies have a predominating effect on each 
market, however. The simultaneous interaction of these twenty-six 
variables produce complex market behavior. This dynamic behavior 
is influenced by past history, time-dependencies and the inter- 


action of all competitors. 


The markets are robust enough to reward a variety of consistent, 
feasible strategies. For example, if firms resort to price cutting 
as a primary tactic, markets become very price sensitive and the 
low price, higher volume, market expands relative to the higher 
priced market. At the other extreme, firms may expand the market 
by increased promotion and R&D expenditures and thus create a 
market which supports both higher and lower priced products. 

Firms can develop competitive positions which emphasize any one 
or a combination of their decision variables and thus attract 


different kinds of customers to the firm. 
Customers 


Each market has two classes of customers who are differen= 
tially sensitized to different combinations of marketing factors. 
Repeat customers are sensitive to pricing, past sales levels and 
inventory availability. Shopper customers are sensitive to the 
relative differences between competitors' price, promotion, R&D 
and inventory policies. Therefore, a firm's share of the market 
is determined by both present and past marketing and production 


strategies relative to those of its competitors. 


The interaction of each competitor's market strategy and 
economic and seasonal conditions cause the relative size of the 
repeat and shopper pools to fluctuate each quarter. Customers 


entering or leaving the market are shopper customers. Once a 
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customer has purchased from a firm, he becomes a repeat customer. 
The majority of repeat customers tend to remain with a firm under 
certain conditions. Repeats, however, can become shoppers but 
may return to the firm if its products are more attractive than 
its competitors. In addition, a firm's ability to attract and 
service customers may experience quarterly changes. Consequently, 


marketing problems continually recur for each firm in the industry. 


Industry 


The industry is composed of six firms. Each firm's operating 
and financial status is identical when the teams assume management 
responsibility in Quarter 12. The operating history for Quarters 
2 to 11 is unique to the firm and was produced by the game admin- 
istrator and his associates, each of whom ran independent firms 
for 11 quarters. To create identical starting conditions for 
Quarter 12, the prior four quarters were collaboratively rerun 


until ending results of Quarter 11 were consistent across all firms. 


The nature of the competition is determined by the inter- 
actions of firms in their quest for customers in both markets. 
The interaction results in complex non-linear reactions in the 
market place and, which, needless to say, are extremely difficult 
to predict. In addition, each firm's actions are not known with 
certainty by its competitors. Needed information with respect to 
the market and competition may not be reported or may be inac- 
curate, if reported. Thus the character of the competition, and 
its present status, are only partially known by each firm. What 
actions firms will take in the future can be estimated only on 


the basis of noisy information, if past history is any guide. 
Firms 


The management of each firm is a four or five man team. The 
Management of each firm controls similar marketing organizations 
located in both the Foreign Market and Domestic Market. Each 
marketing organization sells a product which can be differentiated 


59 


from competitive products by the firm's pricing, promotion and 
R&D strategies in each market. Because the same product is sold 
to both repeat and shopper customers, who are sensitive to 
different product images, decisions in each market require 


thoughtful balance along a number of dimensions. 


The production facility for each firm is located in the 
Domestic Market. The capacity of the facility is a function of 
the effective labor and machine capacities, both of which may be 
increased. Changes, however, require lead times and incur 
significant costs associated with hiring, firing, overtime and 
additions to capacity. Limits on maximum changes permitted in 
any quarter require firms to anticipate their needs substantially 
in advance of requirement. Costs associated with the plant, 
product, selling and administration are both fixed and variable. 
Some of the real costs in each of the areas are unknown to the 


firms. 


The production process is characterized by time delays for 
training workers, installing capacity, manufacturing the product 
and shipping. Goods destined for each market must be designated 
in advance of production since each incur different delays and 
costs in the production process. Goods for the Foreign Market 
are more expensive because of higher labor, materials, overhead 
and shipping costs. Finished goods are warehoused in both 


markets. 


Firms have a number of financial transactions available for 
handling their need for and use of funds. Firms can finance 
operations through debt in either market but can sell stock 
only in the Domestic Market. In addition, funds can be trans- 
ferred between markets and surplus cash can be invested in 
securities in either market. The stock of each firm is sold only 


in the Domestic Stock Market. 
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3.1.2 Nature of Decision Making 


The purpose of this section is to describe the nature and 
inherent difficulty of the decision making of the marketing 
function. This study focused on marketing decisions because they 
involve complex, non-structured decision making. Moreover, 
marketing decisions have counterparts in each market; therefore, 
they fulfill a necessary condition for the experimental design 


and, thus, have implications for validity. 


These five marketing decisions in the Game are shown in 


Figure 3.3. 


Domestic Market Foreign Market 
Domestic Price...........Foreign Price 


Domestic Promotion.......Foreign Promotion 


Research and Development 


Figure 3.3 Marketing Decisions in Game 


Firms recognized the importance of these decisions and 
concentrated most of their time and effort on marketing throughout 
the game. The quality of these decisions is essential to the 
profitability of the firm. Profitability is directly related to 
sales which, in turn, is a function of price, promotion, research 
and development and inventory availability. Although inventory is 
not an explicit market decision, it is influenced by sales and, 


thus, must be considered as an integral part of marketing. 


What, then, is the nature of the marketing decision making 
facing each firm? Stripped to its essentials, the task centered on 
finding a series of sequentially interdependent decisions for 


price, promotion, and R&D with respect to inventories that would 
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maximize the profitability of the firm over an appropriate decision 
horizon. All of that must be done under conditions of highly 
uncertain demand and without all of the necessary information or 


models. 


The complexity of this task is directly related to the nature 
of the markets and stems mainly from its dynamic, non-linear and 
interactive characteristics. First, the markets are dynamic; 
that is, the effects of past decision strategies of all firms 
influence the present decision quarter. Second, the relationships 
among factors which create markets and determine sales are non- 
linear. Parametric curves are not simple linear functions but 
complex relationships. Third, firms in the market are completely 


interactive; the actions of all competitors influence each other. 


The sequential interdependency of the decisions is directly 
related to the dynamics of the system. The set of feasible 
decisions for any quarter rest on the present decision state 
which is a reflection of all past decisions. The results of 
past decisions affect, guide and determine the feasibility of 
later decisions. Moreover, the evaluation of present decision 
alternatives should include consideration of the expected 
consequences in future quarters over an appropriate decision 
horizon. If this is not done, decisions which are optimal with 
respect to the present quarter may be sub-optimal over the longer 


decision horizon. 


The evaluation of consequences is related to the degree of 
understanding or knowledge each firm has about the system. The 
effect of non-linearity either mask interactions or produce 
behavior which appears counter-intuitive. Firms have only partial 
understanding of the impact of the controllable and non-controllable 
variables on the structure and parameters of their decision 
environment. The influence of all the variables, the structure of 
the environment and the relationship represented by the parametric 
curves are difficult to deduce with certainty in either market. 
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The information available for later decisions was contingent upon 
the nature and consequences of previous decisions. Firms rarely 
know the present status of competition because some reported 
information was inaccurate. Therefore, their estimates of what 
competitors have done were somewhat vague. These compound effects 


make understanding the system even more difficult. 


Predicting the outcome of an alternative set of decisions, and 
doing this for a number of periods is difficult, at best. Because 
of the interaction in the market, firms not only had to estimate 
their impact on the market but that of competitors as well. 
Estimates of each competitors' marketing decisions and the pattern 
of economic and seasonal variables for the upcoming quarter and 
predictions of their influence was necessary. This was especially 
important when market demand exceeded inventory available for 
sale in the quarter or, in the longer run, capacity in the industry. 
Particularly in these cases, poor estimation can lead to stockouts 
and substantial losses. Thus, interaction and the inability to 
predict its effects created difficulty in estimating demand and 


making marketing,production,and finance decisions. 


The complexity of the task is increased by the number of 
factors which impacted the market each quarter. From the firm's 
point of view, the size of the total markets and their sales are 
influenced by the interaction of twenty-six variables, only three 
of which are directly controllable by each of the six firms. 
Price, promotion and R&D decisions are made by the firm each 
quarter and are, therefore, controllable decision variables. In 
addition, inventory is only partially controllable since sales 
last quarter and, thus, inventory available for sales this quarter 
are determined not only by the firm's decision but by the actions 
of competitors and the economic and seasonal indices. In short, 
the size, dynamics and interactions taxed the firm's ability to 


deal with marketing problems. 
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3.1.3 Participants 


The purpose here is to supply pertinent statistics on the 
participants in the Game. The participants acted as research 
subjects and, therefore, represent a sample of the real world 


population to which results may be generalized. 


The simulation was designed as an exercise in executive decision 
making for use in graduate and advanced management programs at MIT. 
One of these programs is the Senior Executive Program. This 
program is an intensive, total-learning experience for established 
top-level managers, which includes classes, seminars, and guest 


speakers. 


All twenty-five Senior Executives in the 1972 Spring Session 
of the Senior Executives Program participated in the Game and, 
therefore, in the experiment. 

Pertinent statistics for these Senior Executives are shown 
in Figure 3.4. 


Educational Background 


Doctors: 5 
Masters: 4 
Bachelors: 16 


Professional Affiliations 


Organizations represented: 25 
Positions represented: 23 


Ages within Group 


Range 33-56 
Average 43.7 


Figure 3.4 Profile of Senior Executives 


These statistics briefly summarize more extensive data given 
in Appendix E which suggests the breadth of Senior Executive's 
professional experience, educational backgrounds and management 
positions held at the time of the experiment. The profile which 
emerges is a mature group of professional managers characterized 
by extensive backgrounds, training and experience in a variety of 


positions in production and service organizations. 
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3.2 Interactive Decision Aids 


This section describes each of the interactive decision aids. 
Three systems are available to all firms; a Marketing Decision 
System (MDS), a Financial Planning System(FPS), and a Statistical 
Analysis System (SAS). MDS directly supports the making of 
non-structured price, advertising, and R&D decisions for the 
Foreign Market. FPS helps firms make financial decisions 
concerning cash remittances. securities and loans for either 
market. SAS has capabilities for plotting, regression analysis 
and model building and maintains historical data bases for both 


markets. These systems are an integral part of the MIT Management 


Game. 


The three systems have common characteristics. Each is 
interactive. All firms use the system directly from their hard- 
copy terminals. Each system monitors user input and prompts the 
user when help is needed. All systems have inbuilt tracing 
features which capture user interaction. Although different 
forms of interactive dialogues are used, all systems respond in 
terms familiar to the manager and thus provide capabilities which 


are useful for solving problems and making decisions in the Game. 


To describe every detail would only obscure the essential 
nature of each system. The purpose here is to describe the 
capabilities and use of each system. MDS, the Decision Support 
System, is emphasized because it directly aids marketing decisions 
and its availability in the Foreign Market created the experimental 
condition while its absence in the Domestic Market created the 
control condition for the research design. Since everything else 
is the same for both conditions, differences between DSS-aided 
and non-aided decision making should be attributable to the 


Marketing Decision System. 
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3.2.1 Marketing Decision System 
The Marketing Decision System (MDS) supports decision making 


in the Foreign Market. The system helps firms make marketing 
decisions by providing estimates of how different combinations 
of price, advertising, R&D, and inventory influence the size of 
the market and sales for each firm. The system also aids firms 
in estimating how different economic and competitive conditions 
effect the market. In short, the system assists firms in 
evaluating the consequences of alternative marketing strategies, 


either their own or their competitors' strategies. 


The capabilities listed above, however, fails to capture how 
the system is used to aid marketing decisions. Further elaboration 
is needed; descriptions of the (1) options, (2) conceptual 
operation, (3) inputs and outputs, and (4) system's commands are 
provided below. An example of actual use illustrates the 


capabilities of this system and shows how firms use the system, 


Options 


The Marketing Decision System has two options. the Modeler 
(M) option and the Future (F) option. The Modeler option 
provides sales and market estimates for the next quarter only. 
The Future option provides the same estimates for each sequential 
quarter for any number of periods in the future. Both options 
fully recognizes the dynamic nature of the markets by accounting 
for effects of past decisions and present market status. The 
future option may be pictured as a number of sequentially inter- 


dependent modeler runs. 


Operation 


Conceptually, the system's operation is quite simple. The 
system accepts twenty-six values for input, runs a model of the 
market, and produces estimates of the size of the market and 
sales for each firm. The heart of the system is a model which is 
a structurally perfect representation of the Foreign Market. 
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That is, each variable and relationship is identical to its 
counterpart in the real market. The structure of the market. 
discussed in section 3.1.1 gives the details of this model. 

The quality of the estimates produced by the system, however, 
is highly dependent upon the input values supplied by the user. 


Input 


The user must supply the input shown in Figure 3.5 for every 


quarter simulated by the system. 


Name of Number of Values supplied by user 
Variables Required Values are estimates of: 

GNP 1 Gross National Product 

SEAS 1 Seasonal influence 

PRICE 6 Each firm's price 

ADV 6 Each firm's advertising expenditures 

R&D 6 Each firm's R&D expenditures 

INV 6 Each firm's inventory levels 

SHIP 6 (optional) each firm's additions 


to inventory 


Figure 3.5 Required Inputs for MDS 


The number of required values for each variable is shown above. 
Only GNP and SEAS require one value each while PRICE, ADV., R&D, 
and INV. require six values for each variable, one for each of the 
firms in the market. Each of the twenty-six values must be known 


to the system before the model is run and output is generated. 


Output 


The system's output consists of estimates on the size of the 
total market, firm sales and market share. Two types of output 


are provided: a summary report and a complete report. 
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The summary report is shown in Figure 3.6 


QTR 15 FROM 15 MARKET 2 
TOTAL = 1291088. OUR SALES = 175109. SHARE = 13.55 


Figure 3.6 MDS Summary Report 


This report is automatically printed on the user's console 
after the user requests a run using the CALC command. The quarter 
being simulated and the beginning quarter (in case the Forward 
option is being used) are both identified in the first line. 

The second line gives the estimates for the size of the total 


market, the firm's sales, and the firm's market share. 
The complete report is shown in Figure 3.7 


QTR 15 BEGAN AT 15 
MARKET 2. GNP 206.00. TOTAL UNITS 1326269. 


FIRM UNITS PRICE LOST SALES % THIS 
1 207873. 8.75 1800. 15.7 
2 257621. 8.10 20000. 19.5 
3 258856. 8.50 1500. 19.4 
4 173970. 9.00 8700. 13.1 
5 226308. 8.75 0. 17.1 
6 201641 9.25 90000. 15.2 


Figure 3.7 MDS Complete Report 


This report provides supplementary information for the firm. 
In addition to the information supplied in the summary report, it 
gives estimates of each competitor's sales, lost sales and market 
share. The values for GNP and Price are copies of the input values 
supplied by the user. The form of this report is familiar to the 
decision maker because it is similar to the Market Summary Report 


shown in Appendix B. 
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Commands 


The system is command driven; that is, operations are controlled 
directly by the user. No question and answer dialogue is involved 
unless the user fails to supply necessary values or makes a request 
that the system does not understand. These command names and 


functions are shown in Figure 3.8. 


MARKETING DECISION SYSTEM 


COMMANDS 
COMMAND NAME ACTUAL COMMAND FUNCTION 
BEGIN BEGIN n Simulation begins at period n 
INPUT COMMANDS 
GNP GNP Type to input GNP value 
SEAS SEAS Type to input SEAS value 
PRICE PRICE Type to input PRICE values 
ADVERTISING ADV Type to input ADV values 
RESEARCH & DEVELOPMENT R&D Type to R&D values 
INVENTORY INV Type to input INV values 
SHIPMENTS SHIP Type to input SHIP values 
OUTPUT COMMANDS 
CALCULATE CALC Type to begin simulation 
SUMMARY REPORT SUM Prints summary report 
REPORT REPORT Prints full report 
LIST L Lists all input values 
ADDITIONAL COMMANDS 
HELP HELP Prints console aid messages 
START Sn Moves pointer with line 
PRINT Pn Prints values for variable n 
CHANGE Cnv Changes parameter n to v 
END END Ends input line 
QUIT Q Ends use of MDS 


Figure 3.8 : Marketing Decision Systems Commands. 
Input Command specify the type of values which follow the command. 
For example,to supply prices for each firm the user types 
Price x.xx X.XX X.XX X.XX XK.XK xX.xx end 
Output Commands are used to get results on the console while other 
Commands facilitate input. Use of each Command should be obvious 
from the example to follow. 
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An Example of Use 


An example of actual use is shown here. It is a selected 
reproduction of parts of firm 1's trace. The actual trace contained 
28 runs. This sample illustrates MDS use; comments are provided to 
explain what the user was doing. All upper case lines are system 


responses while lower case lines represent the decision maker. 


LOGIN 
firm 1 *User logs in, selects 
MDS and the MODELER 
PASSWORD option. 
Little 
MDS, FPS, SAS OR LOGOUT? 
mds 
MODELER OR FUTURE OPTION? 
m 
REQUEST? 
1 
QUARTER 15 BEGINNINGQUARTER 15 k ? 5 
GNP 206.00 SEAS HIGH aa aes te 
FIRM PRICE ADV R&D INV previous runs earlier 
1 8.75 30000. 300000. 10000000. incday be the esaten) 
2 8.10 30000. 300000. 10000000. y oY y 
3 8.50 25000. 250000. 10000000. 
4 9.00 30000. 200000. 10000000. 
5 8.75 50000. 300000. 10000000. 
6 9.25 25000. 225000. 10000000. 
REQUEST? 
calc 


QTR 15 FROM 15 MARKET 2 
TOTAL = 1326269. OUR SALES = 207873. SHARE = 15.67 


REQUEST? *User request run and 
summary report and 
report then a complete report 


QTR 15 BEGAN AT 15 
MARKET 2, GNP 206.00 TOTAL UNITS 1326269. 
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QTR 15 BEGAN AT 15 TOTAL UNITS 1326269. 


FIRM UNITS PRICE LOST SALES % THIS 

1 207873 8.75 0. 15.7 

2 258856 8.10 0. 195 

3 257621 8.50 QO. 19.4 

4 173970 9.00 0. 1361 

5 226308 8.75 0. 17.1 

6 201641 9.25 0. LS «2 
REQUEST? 
PRICE 9.50 end *User tests range of prices 

in the following sequence 

REQUEST? 
calc 


QTR 15 FROM 15 MARKET 2 
TOTAL = 1292088. OUR SALES = 175109. SHARE 


13.55 


REQUEST? 

begin 15 

REQUEST? 

price 9.25 end 

REQUEST? 

calc 

QTR 15 From 15 Market 2 

TOTAL = 1269604. OUR SALES = 182324. SHARE = 14.36 


begin 15 
REQUEST? 


price 9.00 end 


REQUEST? 

calc 

QTR 15 FROM 15 MARKET 2 

TOTAL = 1274800. OUR SALES = 190398. SHARE = 14.94 
® 
@ 
@ 


*The abbreviated sequence above illustrate how MDS 
was used to aid in the pricing decision. The 

trace showed 9 prices were tested in this sequence. 
The user then turns his attention to the advertising 
decision. e 


@ 
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REQUEST? 


Begin 15 | *User sets price to the 
original 8.75 and tests 

REQUEST? values for advertising. 

Price was then incremented 

price 8.75 and various advertising 
levels retested. 

REQUEST? 

adv 60000 

REQUEST? 

calc 


QTR 15 FROM 15 MARKET 2 
TOTAL = 1348265. OUR SALES = 222801. SHARE = 16.53 


REQUEST? | 
begin 15 


REQUEST? 
adv 30000 end 


REQUEST? 

calc 

QTR 15 FROM 15 MARKET 2 
TOTAL = 1329892. OUR SALES 209993. SHARE = 15.79 


*This illustrates how price/inventory combinations 
were tested. The user tested 6 combinations in 
the sequence from which the sample was taken. 

The user now turns his attention to R&D. 


@ 
@ 
REQUEST? 
begin 15 
*User resets price to 
REQUEST? original level and 
price 8.75 begins to test price/ 
R&D combinations. Price 
begin 15 is then incremented and 
REQUEST? other levels of R&D tested. 
-r&d 275000. end 
REQUEST? 
calc 


QTR 15 FROM 15 MARKET 2 
TOTAL = 1277660. OUR SALES = 198236. SHARE = 15.52 
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begin 15 
REQUEST? 
r&éd 325000 end 


REQUEST? 

calc 

QTR 15 FROM 15 MARKET 2 

TOTAL = 1286043. OUR SALES = 202848. SHARE = 15.77 


REQUEST @ 
@ 
@ 


*At this point, the user had completed his sensitivity 
studies. Having decided tentatively what his decision 
might be, he now begins to confirm his final determination. 


e 
@ 
@ 
REQUEST? 
GNP206.00 SEAS high 


REQUEST? 
price 9.15 8.10 8.50 9.00 8.75 9.25 end 


REQUEST? 
r&d 300000 250000 225000 250000 300000 225000 end 


REQUEST? 
adv 45000 60000 30000 30000 50000 25000 end 


REQUEST? 
inv 195000 250000 250000 175000 200000 175000 end 
REQUEST? 


OUARTER 15 BEGINNING QUARTER 15 
GNP 206-00 SEAS HIGH 


FIRM PRICE ADV. R&D INV 
1 9.15 45000. 300000. 195000. 
2 8.10 60000. 250000. 250000. 
3 8.50 30000. 225000. 250000. 
4 9.00 30000. 250000. 175000. 
5 8.75 50000. 300000. 200000. 
6 9.25 25000. 225000. 175000. 
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REQUEST? 

begin 15 

REQUEST? 

calc 

QTR 15 FROM 15 MARKET 2 

TOTAL = 1245000. OUR SALES # 195000. SHARE =1566 


QTR 15 BEGAN AT 15 
MARKET 2, GNP 206.00 TOTAL UNITS 1245000. 


FIRM UNITS PRICE LOST SALES % THIS 
1 195000. 9.15 1806. 15.7 
2 250000. 8.10 -17922. 20.1 
3 250000. 8.50 11445. 20.1 
4 175000. 9.00 5116. 14.1 
5 200000. 8.75 28236. 16.1 
6 175000. 9.25 29364. 14.1 

begin 15 

REQUEST? 

price 9.25 end 

REQUEST? 


Inv 195000 250000 250000 175000 200000 175000 end 
REQUEST? calc 

calc report 

QTR 15 FROM 15 MARKET 2 

TOTAL = 1245000. OUR SALES = 195000. SHARE = 15.66 
QTR 15 BEGAN AT 15 

MARKET 2, GNP 206.00 TOTAL UNITS 1245000. 


FIRM UNITS PRICE LOST SALES % THIS 
1 195000. 9.25 0. 15.7 
2 250000. 8.10 17779. 20.1 
3. 250000. 8.50 11345. 20.1 
4 175000. 9.00 5033. 14.1 
5 200000. 8.75 28124. 16.1 
6 175000. 9.25 29289. 14.1 
e 


*At this point, the user began to use the Future option 
to evaluate the future effect of his present decision. 
Since the trace is similar, these runs are not shown. 
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3.2.2 Financial Planning System 


The Financial Planning System (FPS) produces pro-forma 
financial and operating statements given estimated sales in the 
Foreign and Domestic Markets and the decisions of the firms. The 
system is based on a perfect model of the firm. The simulation 
accurately predicts a firm's status for an upcoming quarter for 
any set of decisions and expected sales levels. The system 
supports financial decision making in both markets by determining 
profit,cash and operating positions for the firm. It helps 
firms . determine the proper cash, securities and loan balances 
for the next quarter of operation. To make these decisions several 


analyses can be run simultaneously to compare resulting differences. 


The Financial Planning Systems has two options; the Planner (P) 
option and the Forward (F) option. The Planner option allows firms 
to simulate one period in the future. The Forward option allows 
firms to simulate operations for any number of sequential periods 
in the future. The interactive dialogue for both options is 


similar. 


Several Planner and Forward runs can be initiated at one 

time by the firms simply by supplying the required input which 
consists of eighteen values: 

(1) an estimate of domestic sales 

(2) an estimate of foreign sales 

(3-18) sixteen decisions of the firm 
This input may be entered in either of two modes. In the request 
mode, the system prompts the user for each line of input. In the 
accept mode, the user enters all required data without interruption 
from the system. In either mode, however, the system monitors 
input and automatically requests missing information. Additional 
commands are available for printing input data on the console and 


for changing values prior to running the simulation. 
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Once input is complete, the system determines financial and 
operating status for the firm in a form identical to the reports 
normally received as a result of actual operations. These ten 
reports include foreign and domestic: 

(1) Balance Sheets 

(2) Income Statements 

(3) Sources and Uses of Funds 

(4) Reconciliations of Retained Earnings 
(5) Reconciliations of Inventory 


A copy of the actual reports is shown in Appendix C and D. 


Output is requested by the user by specifying line items or 
special reports to be printed on the consoles. Since the complete 
reports listed above for either market require four pages of 
output, special commands allow firms to select specific line items 
from any report and have them printed on their console. To print 
special reports requires entering only the report number shown 
below. Five special reports were produced by the system. These 
consisted of selected combinations of line items of particular 


interest to firms. These reports are: 


Report Number Contents 
1 A statement of net income after tax, net 


change in cash position, goods available 
for sale for both markets. 


2 A manufacturing analysis of the overhead, 
overtime, labor, hiring and firing and 
reshipment expenses. 


3 An abbreviated profit and loss statement 
for the Domestic Market. 


4 An abbreviated profit and loss statement 
for the Foreign Market. 


5 A statement of cash, securities, and loan 
positions for both markets. 

To summarize, this system aids all firms in making financial 
decisions. It helps firms to determine the financial and operating 
consequences of different sets of decisions and sales forecasts and, 
thus, has the potential for improving the utilization of financial 


resources. 
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3.2.3 Statistical Analysis System 


The Statistical Analysis System (SAS) assists decision makers 
in performing regression analyses and building models. The 
system's capabilities are all intimately related to the data bases 
maintained by the system. The data base contains information on 
both markets. These bases are automatically updated by the system 
as the Game is run. In addition, firms can specify their own data 


elements and transformations to the system. 


The system capabilities include plotting, regression analysis 
and model building. Plotting routines allow the user to print 
Scattergrams and time series plots on the console. The multiple 
linear regression package supplies complete statistics for the 
regression equation. Statistics included means and deviations 
for all variables, pairwise correlations, analysis of variance 
and significance tests. The model building capabilities allow 
users to define their own expressions and transformations based 
on any data elements. User supplied transformations become part 
of the system maintained data bases and are available for 


modification or regression analysis. 


The system prompts users through a series of questions while 
monitoring all responses and data input. This interactive 
dialogue allows the user to specify exactly what he wants done. 
The system automatically performs the necessary calculations and 
prints the requested output on the console. In short, the 
system's output is useful for making marketing decisions. It 
helps firms develop models for predicting the size of the 
market and their market share. These models have the potential 
for aiding decision making, especially in the Domestic Market 


since no other formal support is provided for this market. 
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3.3 Implications for External Validity 


This section discusses the implications for external validity 
which relate to the nature of the simulator, the decision aids and 


the participants in the Game. 


External validity centers on the degree of confidence that 
laboratory results match what would occur in the world at large. 
The ability to generalize from one to the other is a question that 
can never be answered with certainty. Further proof is required in 
the laboratory and,more importantly, in the field. Campbell and 
Stanley (20 ) underscore the necessity of repeating experimentation 


in order to establish confidence. 


Hopefully, the knowledge gained in the laboratory is a good 
indication of what will be found in the real world. The quality 
of experimental results rest on the foundation of high internal 
and procedural validity. If high internal and procedural validity 
are present in the research design, external validity is mainly 
dependent upon the degree of representativeness of the laboratory 
counterparts of the environment, conditions and population to 
which results may generalize. These correspond to preceding parts 


of this chapter in the following manner. 


(1) Environment - Nature of Game Environment 
(2) Conditions - Nature of Decision Task and Aids 
(3) Population - Participants in Game 


This framework will be used to structure the evaluation of 


external validity for this research. 
3.3.1 Issues Related to Environment 


The environment used in this study was not the real world; 
"reality" was simulated by the MIT Management Game. Even though 
the environment was simulated, the potential of games for 
experiments having high external validity has long been recognized. 
Shubik ( 99) notes that computer-based management games have greater 
potential to represent the complexity found in the real world than 


78 


do the more traditional experimental situations. Chapanis (23 ) 
asserts that games contrast sharply with the sterile, controlled 


environments which characterize most decision making research. 


The central issue in the use of simulation as a research 
vehicle is whether the degree of realism and complexity in the 
simulated environment is representative of the real world. 

Raser (87 ) asserts that the concepts of realism and complexity 
are more appropriate for judging the validity of a game for 
research purposes than the traditional criteria of isomorphism of 
the model and reality. Shure, Rodgers and Meeker (101) rightly 
caution against the temptation to judge validity solely on the 


basis of isomorphism to the reference system. 


Realism and complexity can be evaluated from different 
perspectives, for example, from actual similarity to a referent or 
perceived similarity to a situation. Actual similarity is usually 
measured by isomorphism of the simulation to the real world. 

Of greater relevance to the present study, however, is the 
perceived similarity of environment. Hedburg (53 ) maintains 

that the perceived semblance between the simulation and its 
counterpart are more crucial than the actual similarity because the 


research subjects responses will automatically be more natural. 


The environment should be sufficiently realistic so that 
decision processes normally used in the real world would be only 
minimally distorted. Moreover, the environment should be complex 
enough to provide problems that challenge the capability of the 
decision maker. Goffman ( 46) suggests that involvement is not 
a function of objective reality but rather the psychological 
reality of the situation. The crux of the situation is whether 
or not the simulated environment creates the psychological 


equivalent of reality for the decision makers. 
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The realism of the environments in the MIT Game was suggested 
by description of the Foreign and Domestic Markets. These markets 
are dynamic, non-linear and interactive. Each was influenced by 
many (26) variables, most of which were non-controllable from the 
firm's point of view. The inherent complexity of the problems 
facing each firm were determined by the nature of the markets. 
For example, to "understand" the nature of the markets or to 
estimate the status of competition was difficult. Each firm 
influenced both markets, but the character of each was shaped by 
the interaction of all competitors and the economic variables. 
These characteristics are found, as Forrester ( 42 ) contends, in 


real world situations faced by practicing managers. 


The degree of realism and complexity perceived by the partici- 
pants can be evaluated by their response to the questionnaire and 
comments during the interview.’ Figure 3.9 show perceived realism. 


(Questionnaire, item ) 
How realistic was the marketing environment in the game? 


Responses 


Extremely Very Fairly Not 
realistic realistic Realistic realistic realistic 


Figure. 3.9 Perceived Realism of Environment 
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If no decision aids were provided, how complex were the marketin 
problems in the game? 


Extremely Very Complex Fairly Not 
complex . complex complex complex 


Figure 3.10 Perceived complexity of marketing problems. 


The interview permitted more open-ended responses and served as 
a check on the questionnaire items. A range of responses is reported. 


(Interview, items ) 
Do you (team members) feel that the Foreign and Domestic Markets 
were realistic? 
"Yes, it was damn realistic - it certainly kept me intrigued" 
"Maybe its a game but I thought it was quite realistic" 
"T'm sure we didn't understand how the markets work and that is 
realistic in my business" 
"Well, maybe, but it’s certainly not the real world - Fairly 
realistic I'll say" 


How complex were the problems you faced in each market? 
"Too complex" 
"Ha, those markets were bears" 
"TI felt comfortable - maybe lost is a better word" 
"Those damn markets were tough -— complex enough in my book" 
"They 're.no where near as complex as the real world' 


Viewed as a whole, the responses to the questionnaire and the 
interview indicate that the environment and the associated decision 
problems were perceived as both fairly realistic and complex. The 
ongoing decision environment was, in large measure, the psychological 
equivalent of reality for the participants. Participant responses 
should be representative in this environment. 
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3.3.2 Issues Related to Conditions 


Related to the realism and complexity of the environment is the 
representativeness of conditions created in the study. The 
generalizability of conditions depend upon the similarity of 
circumstances between the laboratory and its real world counterparts. 
The critical question is: to what specific situations can results be 
transferred? Among the relevant considerations here are the (1) 
representativeness of the task and decision making behavior and 
(2) the quality of information and (3) decision aids. In other 
words, is each of these similar to what might be found in real 
world situations. 


Decision Task and Processes 


Briefly, the decision task involved making a set of sequentially 
interdependent decisions to meet certain profit objectives under 
conditions of limited information and decision support in a 
dynamically uncertain environment. Forrester (42 ), Moskowitz (74 ), 
and Conrath (27 ) have noted that these characteristics are found 
in many actual decision making situations. Powers has remarked: 


"The task of a decision maker in "real-world" 

systems is that of making a number of decisions 

sequentially in time, where the outcomes and 

payoffs of earlier decisions affect, guide, and 

serve as inputs to the making of later decisions. 

The objective is usually the maximization of 

total profit over the interdependent sequence | 

of decisions." 
Trull (113claims that the difficulty of handling these tasks stems 
from the lack of a measure or index of effectiveness, coupled with 
a surfeit of alternatives and an absence of reliable probabilities 
associated with decision variables. At the very least, decision 
tasks in the game are inherently difficult and may resemble those 
encountered by the professional manager. Certainly, task 
complexity far exceeds what is found in traditional laboratory 


studies. 
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Coping with the decision task shapes the decision processes. 
The decision making studied here involved consideration of a large 
number of alternatives involving many controllable and non- 
controllable variables whose identification and evaluation was 
difficult, at best, given the uncertainty in the environment, and 
the multiple, conflicting goals and lack of adequate formal methods 
or measures of effectiveness. These features are similar to (108) 


Soelburg's characterization of non-structured decision making. 


This type of decision making has been labelled non-programmed 
by Simon (104). Scott Morton ( 93) notes the nature of non- 
structured problems as: 


"the problems were ill~structured, hard to find, 
and once found, hard to define in operationally 
useful terms." 


In much the same terms, Mason and Mitroff ( 65) characterized 
these problems as "wicked" and expressed the conviction shared by 
many Management Scientists (1 ) (33 ) that: "real management 
problems appear overwhelmingly wicked or ill-behaved." Ebert (35 ) 
suggests that decision making similar to the marketing decisions 
in this game may be characteristic of a much broader group of 
decision problems classified as dynamic decision problems under 


conditions of uncertainty. 


Quality of Information 


The, quantity and quality of information on which decision 
making was based are important considerations in determining the 
representativeness of conditions. Several responses during the 
interview testified to the fact that the quantity of information 
(over 250 information items each quarter) tended to overload 
managers in some cases. However, the problem was not of particular 
importance; the information was available, but its use was left 


to the discretion of the decision maker. 


Of more importance was the quality of the marketing information 
provided in the "Market Summary" report shown in Appendix B which 
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firms received every quarter. This report was designed to simulate 
the quality of marketing information that might be found in the 
real world. Therefore, information pertaining to projections of 
GNP, sizes of total markets, actual and lost sales, promotion, 
research and development expenditures were reported with varying 
degrees of "noise" in much the same way that industry statistics 
would be. The quantity and quality of reported information were 


identical for each market. 


Certain information which the participants believed would have 
been helpful was not available. For example, only total sales 
were reported. Firms had no way of knowing their relative number 
of repeat and shopper customers. Information was lacking on 
seasonal influences and exact breakdowns of inventories were 
unknown. Since the companies were considered publicly held, the 
"Market Summary" contained accurate information of financial status 
for all companies but only minimal information was available 


concerning operations. 


The quantity and quality of information were considered 
similar to what participants normally received on the job as 
evidenced by responses in the interview: 


"TJ felt comfortable with it - the information. It would 
have been a lot easier without the noise but that's 
pretty realistic" 


"Same kind of information I get at work although those last 
sales reports were really off" 

"we should have known how many different types of customers 
(repeats and shoppers) we got each quarter" 

"I would have liked more information - accurate information- 
hell - sometimes we were only guessing - we never really 
knew what competition was doing" 

"Too damn much - I bet we only looked at 5% of all that 
stuff - those "market summary" reports, that's where the 
market information is - the firm reports...well, I really 
wasn't that concerned with them" 
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Interactive Decision Aids 


The support provided by the interactive decision aids relates 
to the representativeness of the situation. Surveys (17 ), (14 ), 
( 44) indicate that interactive support of this kind is not 
prevalent in the real world at present, but may become more avail- 
able in the future. In this sense, the decision aids created an 
atypical situation. The point to note is that interactive programs 
may represent what might be commonplace in the future. The 
remainder of the discussion should be considered from this 


perspective. 


The Financial Planning System (FPS) produced pro forma 
financial and operating statements based on anticipated Foreign 
and Domestic sales and the decisions of the firm. Since the 
necessary calculations were known to the participants, anyone 
could have produced the same results, although much more time and 
effort would have been required. Even though the aid provided by 
FPS was not unusual, the participants valued it highly. Some 
participants indicated that similar aids were available in their 
companies or that the service was provided by the accounting 


function. 


The Statistical Analysis System (SAS) allowed the decision 
maker to access, print and graph data, perform regression analyses 
and build models. It is doubtful whether regression analyses 
could have been carried out by most of the participants without 
the aid of a system. Many participants indicated, however, that 
their companies had similar capabilities available. Senior 
Executives, on the whole, did not use these services themselves 


but instead relied on the staff if this type of analyses was needed. 


The Marketing Decision System (MDS) was a true Decision Support 
System. It allowed the decision maker to interact directly with a 
simulation model of the Foreign Market. Certainly, the interactive 
interface, access to data and computational support is represen- 


tative of the better Decision Support System while the quality of 
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the model may be superior to models in the real world. In this sense, 
MDS was atypical. The quality of output, however, depended upon use. 
The system could produce deceiving results if input estimates supplied 
by the decision maker were not of high quality. Responses to the 
questionnaire indicate the decision makers' perceptions of the quality 
and usefulness of MDS and are shown in Figure 3.11 and 3.12. 


(Questionnaire, items ) 


Estimate the quality of MDS as a model of the Foreign Market. 


Responses. 


Extremely Very Good Fair Poor 
good good 


Figure 3.11Perceived quality of MDS 


Estimate the usefulness of MDS for the Foreign Market. 


Responses 


Extremely Very Useful Fairly Not 
useful useful useful useful 


Figure 3.12Perceived usefulness of MDS 
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It is interesting to note that the usefulness of the system 
was rated higher than the quality of the model. In the interview, 
firms clarified their response to these questionnaire items. 

A sample of the answers is given below. 


"Tt certainly was useful for us - the model could be 
improved, it requires too many variables for a run" 
"The system?, it was useful - highly useful- the model- 
not so good, it didn't predict exactly, never did" 
"MDS was really useful - so I guess its a good model" 
"T thought it was a good system but you had to use it right" 
"Why the hell didn'twe have an MDS for the Domestic Market" 
"There is no question that the system was very useful - 
it saved us in the Foreign Market - I would like to have 
the same thing back home...the quality of the model was 
O.K., I think, but its hard to be sure” 


No participant had a similar system available on the job, nor 
did many know of any similar systems. The majority, however, were 


interested in the possibility for developing such systems. 


The results of this study may be generalizable to those 
conditions in which the manager is confronted with decision tasks 
and processes of similar or lower complexity than those discussed 
here. The quality and quantity of information may be a facsimile 
of existing conditions in the real world, while the nature of the 


decision support may be more representative of the future. 
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3.3.3 Issues Related to Participants 


Two critical issues influence external validity with respect 
to Participants : first, the degree of representativeness of 
the participants to the larger universe and, second, their 


motivation in the game. 


The participants in the present game-experiment were all 
Senior Executives at the Sloan School of Management at MIT. 

The extent to which these managers are representative of a 
larger population of higher-level executives relates to the 
generalizability of this research. Certainly, there are few 
reasons to believe that they are non-representative in this 
respect. To the contrary, this group of Senior Executives is 
more representative of managers in the real world than are some 
of the groups used in previous studies of decision making. 
Dill's ( 34) classic remark is appropriate: 

“what college sophomores do, alas, may not 

be much more relevant than the behavior of 

monkeys for predicting how executives, nurses 

and research scientists will perform." 

The profile of the group- its diversity - suggests other 
implications. In many studies reported in the literature, 
experimental subjects have had similar educational backgrounds, 
job positions or experience. For example, Gerrity (45 ) focused 
on portfolio managers. Hedburg studied twenty year old graduate 
students and thirty year old bank executives. Ebert (35 ) and 
others (58 ) (75 ) (116) used undergraduate students as 
experimental subjects. The Senior Executives participating in 
this study, however, represent diversity along a number of dimensions 
as suggested in Appendix E . The implication is that results may 
not be limited to groups with specific characteristics but, in 


fact, may be applicable to a wider context. 


The ideal group of subjects are those selected from the 
population to which the researcher would like to generalize. The 
target population of interest is the group of higher-level executive 
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who make key decisions in the real world. This study was extremely 
fortunate to have had the cooperation of Senior Executives as 
research subjects; to some extent, they may be representative of 
the universe of professional managers who may use Decision Support 
Systems in the future. Limitations should be noted, however. 

This group of managers may be atypical in the sense that they 
received seven weeks of extensive training prior to the experiment 


and that this select group was able to attend the course at MIT. 


The issue of participant's motivation has important implications 
for representativeness of the behavior exhibited by the decision 
makers. Raser (87 ) suggests that motivation is related to realism 
in that natural rather than staged responses are necessarily 
desirable and that the degree of challenge should intrigue the 
participants. Certainly, game involvement is not necessarily as 
intense as real life involvement, nevertheless, it is equally 
foolish to ignore the evidence that games are usually more 
involving than frequently assumed and that a great many "real life" 
situations are not particularly engaging. Anyone who has conducted 


complex games can testify to the intense involvement of most players. 


The involvement of this group of Senior Executives in the game 
was interesting. The game culminated their nine week program. 
Extremely conscientious, this class was characterized by one 
administrator as "one of the best Senior Executives group ever’. 
The experiment provided the opportunity to test new-found skills. 
From all outward appearances, the group was naturally competitive 
and the game offered an environment for matching wits with class- 
mates. Scheduled events were not missed by either Senior 
Executives or the Game Administrator; the game ran smoothly and 
this tended to keep interest high. In the interview, participants 
reported that they viewed the game as a learning process and 


relished the exposure to computer-based decision support. 
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Statistics of attendance may be indicative of the motivation 
of participants and, therefore, involvement in the experiment. 
Since the Game lasted over a period of three weeks, several related 
measures are shown below to indicate the degree of involvement of 


the experiment. 


Attendance at console sessions is shown in Figure 3.13. 


Number of men per session} 


8 Firm 1 
2 
1 
4 
3. Firm 2 
2 
1 
L 4 
3 Firm 3 
2 
1 
4 
3 Firm 4 
2 
1 
L 4 
3 Firm 5 
2 
1 
4 
3 Firm 6 
2 
1 


12 13 14 15 #16 #17 18 19 «——— Quarter 


Figure 3.13 Statistics of Attendance 
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Furthermore, Senior Executives indicated high involvement by 
the number of hours spent on the game in addition to console 
sessions. Figures 3.14 to 3.17 gives the time distributions. 
Questionnaire, items  ) 


Please estimate as accurately as possible the number of hours in 
addition to scheduled console sessions which you spent in game 


related activity. 


[to ee ae 


15 
10 
5 
0 2 4 6 8 10 4 6 8 10 12 14 
Pregame weekend hours First game-week hours 
Figure 3.14 Figure 3.15 
15 
10 
5 
0 2 4 6 8 10 4 6 8 10 12 14 
Intragame weekend hours Second game-week hours 
Figure 3.16 Figure 3.17 


Figures 3.14 to 3.17: Additional hours devoted to Game 


excluding 24 hours of console sessions 


This data suggest that Senior Executive voluntarily spent 
considerable time on game-related activity, both in the scheduled 
console sessions and during their free time. This implies that they 


were naturally motivated to participate in the experiment. 
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3.4 _Chapter Summary 


This chapter documents the design of the Game and the interactive 
decision aids and explores implications for external validity. The 
structure of the Game creates a realistic decision environment which 
is similar to real-world situations. The nature of the non-structured 
decision making simulates the complexity of actual managerial decision 
making. The complexity of the decision making environment in the Game 
far exceeds what is found in traditional laboratory studies, 

The use of Senior Executives as research subjects is in sharp 
contrast to the usual subjects for experimental research, All of these 
factors contribute to the high generalizability of this research. 

The nature and use of the interactive decision aids, MDS, FPS, 
and SAS is shown by console trace. The trace illustrates how firms 
used the system in making the non-structured marketing decisions. 

Issues related to external validity are resolved using question- 
naire data. The data suggest that the environment and associated 
decision making was perceived as highly realistic, task complexity 
was formidable, and that the quality of information was a facsimile 
of actual conditions. Consequently, the results of this study are 


generalizable to similar conditions in the real world. 
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CHAPTER 4 


DESIGN AND CONDUCT OF THE EXPERIMENT 


The design and conduct of an experiment determines the internal 
validity of a study. Better experimental design controls factors 
which jeopardize validity. What can threaten validity is the 
possibility that other variables are responsible for observed 
results. The design used here and the way in which the experiment 
was conducted control factors that have potential for influencing 


results. Consequently, the study has high internal validity. 


This chapter provides detailed description of the design 
and conduct of the experiment and evaluates its inherent validity. 
The organization and contents of the chapter are shown in Figure 4.1. 
4.1 Design of the Experiment 


4.1.1 Structure of Experimental Design 
4.1.2 Associated Design Considerations 


4,2 Conducting the Experiment 


4.2.1 Nature of Experimental Procedures 
4.2.2 Associated Experimental Conditions 


4,3 Implications for Internal and External Validity 


4.3.1 Issues Related to Similarity of the Markets 
4.3.2 Issues Related to Design Experiment 
4.3.3 Issues Related to Conducting the Experiment 


4,4 Chapter Summary 
Figure 4.1 Contents of Chapter 


The design of an experiment should be distinguished from the 
conduct of that experiment. Design translates goals into an 
experimental plan. Conducting the experiment involves implementing 
this plan. Recognizing this, the first section lists the goals that 
guided the design and describes every aspect of the experimental 
design. The second section describes the way the experiment was 
conducted and data collection procedures, The third section 
discusses major issues related to the design and conduct of the 
experiment in order to evaluate the validity of the design. 
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4.1 Design of the Experiment 


This section describes the design of the experiment - its 
structure and related design considerations. The experimental 
design presented here may be more complex than the more traditional 
designs in the literature (e. g. Chapanis(23), Kerlinger (59), 
Campbell and Stanley (20)). The design is easier to understand if 
some of the original goals are kept in mind. These goals are 
reviewed here; they served as guides for the design of this 
experiment. 


(1) Comparative study of aided and non-aided 
decision making 

(2) Adequate experimental control of all variables 
(3) Simultaneous exposure to experimental conditions 
(4) Parallel observations 

(5) Unobtrusive data collection 

(6) Replication of the experiment 

(7) Reliable measurement procedures 

(8) Realistic and complex experimental environment 
(9) High generalizability 

(10) Range of non-structured decision making 

All of these goals increase validity and are related to the 


design and conduct of the experiment. 


4.1.1 Structure of the Experimental Design 


Experimental designs are the specified arrangement of conditions 
that produce data and are characterized by: 
(1) Nature of the experimental environment 


(2) Nature of the experimental conditions 
or treatments 


(3) Assignment of subjects to experimental 
and control groups 


(4) Character of the observations 


Taken together, these features determine the structure of any design. 


The structure of the experimental design used in this study is 
shown in Figure 4.2. Certain conventions make the diagram easier 


- to understand. The vertical dimension indicates temporal order while 
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Game 
Foreign Market _ Domestic Market 


Gaming Exercise 


Run for 8 Quarters 


Firm Reports 


Figure 4.2 Design of the Experiment 
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the horizontal dimension indicates simultaneous occurrence of events. 
The features of the design and issues related to validity are 
discussed in later sections. The immediate task is to explain: 

(1) how the game was used to create the decision environment, 

(2) how the experimental conditions were created, (3) how the 
participants were assigned to groups and (4) how the data was 
collected. To do this each part of the diagram is reproduced in 


greater detail and explained below. 
Decision Environments 


The MIT Management Game was used to create structurally 
identical decision environments for the Foreign Market and the 


Domestic Market as illustrated in Figure 4.3. 


Management Game 


simulates structurally 
identical decision 
environment 


Domestic 
Market 


| 
| 
| 
y Figure 4.3 Decision Environments 4 


For purposes of the experiment, the Foreign and Domestic 
Markets were generated by the same model; only constants and 
multipliers for GNP and the breakpoints for normal values were 
dissimilar. The variables, structure and parametric curves were 


identical. In fact, the same computer code generated both markets. 


Consequently, this design created conditions in which teams 
competed in two markets which were structurally identical except 
for size and growth rates. Thus, the nature of the decision 
environment and the task complexity is, for all intents and 
purposes, quite similar, if not identical. The reason that 


environments must be similar is that it allows parallel observation 
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and use of firms as their own control. For these two conditions 
to hold simultaneously, the decision environments must be as 
similar as possible. All of these conditions, which were created 
by the researcher, were unknown to the Senior Executives who 


participated in the experiment. 
Conditions 
The conditions created by this design are illustrated in 


Figure 4.4. 


Cc 


Foreign Domestic 
Market Decision Aids Market 


Assigned to each 
Market 


FPS 


I 
| 
i 
i 
' MDS SAS 
| 
i 
| 


Marketing 


Decision 


System 
MDS 


i 
} 
1 
i 
v 

Experimental Condition Control Condition 

Aided Decision Making Non-Aided Decision Making 


Figure 4.4 Experimental and Control Conditions 


The experimental conditions were created by providing a 
Marketing Decision System (MDS) to aid marketing decision making 
in the Foreign Market while simultaneously prohibiting its use in 
the Domestic Market. Therefore, the non-structured marketing 
decisions were directly aided by MDS in the Foreign Market while 


their counterparts in the Domestic Market were not. 
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In addition, a Financial Planning System (FPS) and a 
Statistical Analysis System (SAS) were available for use. 
FPS primarily assisted financial planning but had little or no 
impact on marketing decisions. SAS, on the other hand, had the 
potential for aiding marketing decisions in either market through 


the development of models, but was rarely used. 


Technically, decision making in the Domestic Market can not 
be considered non-aided because firms used SAS to develop simple 
models in both markets. For purpose of this study, however, the 
Domestic Market is considered non-aided because no direct decision 
support was provided by the condition created in the experiment. 
The distinction is made because MDS has some of the characteristics 


of a true Decision Support System while SAS does not. 


Figure 4.4 shows that the following terms obviously refer 
to the same condition and can be used interchangeably depending 


upon the emphasis desired. 


Foreign Market Domestic Market 


Aided Decision Non-Aided Decision 


Making Making 
Experimental Control 
Condition Condition 


Figure 4.4 Comparative Experimental Conditions 


Groups 


Participants were assigned to six firms before the Game 
started. Figure 4.5 shows the participant assignment and 
indicates that each firm competed simultaneously in the 
Foreign and Domestic Markets. Since the same firms competed 
simultaneously in both markets, they acted as their own control. 
This is in sharp contrast to the usual situation in which subjects 
are assigned to an experimental and control groups which consist 


of entirely different people. 
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Figure 4.5 Participant Assignment 


The twenty-five Senior Executives were assigned to the six firms 
by the Senior Executive Program Administrator. Since the game was 
played late in the program, he was conversant with the backgrounds, 
talents and interests of the participants. Every effort was made 
to use generally accepted procedures of gaming to balance each 
firm with respect to these dimensions. The figure shows that the 
experimental "subjects" in this research were six firms. Each 
firm consisted of a four-man team since one man was unable to attend, 


thereby reducing the one five-man team to four men. 


Specific organizational assignments were not made by either 
the program or game administrators or by the experimenter. Some 
differentiations in terms of responsibility for functional areas 
occurred on the basis of expertise and interest, however. Since 
each man participated in the same program of study at MIT and, 
moreover, was capable of handling any functional area, teams as 
a whole, concentrated their main efforts on the critical decisions 


in the marketing area. 
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Observation 


This experiment involved comparison of aided and non~aided 
decision making. Figure 4.6 indicates that the game was run for 
eight quarters and permitted simultaneous observation of actual 


decision making for both conditions over these periods. 


Foreign Domestic 
Market Gaming Exercise Market 


Run for 8 Quarters 


| 
| 
| 
| 
| 


Firm Reports 
Qtr. 12 


Firm Reports 


Figure 4.6 Actual Results Produced by Game 


The experiment provided quarterly data on actual results - 
all firm and administrator's reports. An important point is that 
observation and data collection was unobtrusive because the researcher 
acted as an assistant to the Administrator and therefore, had access 
to all results from the Game. Additional data was produced by 


procedures described in the next section. 
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4.1.2 Associated Design Considerations 


Several associated design considerations which characterize 

this experiment include factors related to (1) the firms and 

(2) the research situation. From the researcher's standpoint, 
some of these factors were controllable, while others were not. 
The total number of participants, team assignment, and the game 
schedule were dictated by the situation - the ongoing Senior 
Executive Program - rather than by the experimental design. In 
other words, the experimenter had to work within the confines of 


an established gaming exercise. 


Factors Related to Firms 

The firms were teams of Senior Executive decision makers. 
Because the assignments were made by the Program Administrator, 
the researcher had no control over the total number of participants 
or their assignment to teams. Fortunately, four men were assigned 
to each firm. Prior management gaming exercises had shown that 
this was an adequate number of decision makers for each firm. 
Even though participation in the experiment was voluntary, every 


Senior Executive was highly cooperative. 


The level of computer expertise was approximately equal across 
the six firms. A survey by the researcher revealed that only one 
decision maker in one firm had substantial skills in computation 
but even he did not have extensive exposure to interactive decision 
aids. In contrast, other Senior Executives had little expertise 


in computation or in using Decision Support Systems. 


Every Senior Executive was exposed to similar conditions during 
the experiment. For purposes of the experiment, car pools were 
reassigned (for travel between Endicott House and the MIT campus, 
one hour trip each way) so that firms would have that opportunity 
to discuss the days events. This proved effective; firms reported 


the game was the focus of interest over the two-week exercise. 
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Factors Related to the Situation 


The researcher had more control over the experimental situation 
as most factors were related to the simulation. The starting 
conditions for each firm were equal; that is, each assumed 
management of an identical firm in Quarter 12. The quality and 
quantity of information reported each quarter was equal for all 


firms during the entire experiment. 


The game schedule, however, was developed in conjunction with 
the Senior Executive Program Administrator. Similar to the 
schedule for previous gaming exercises in the Program, the game 
consisted of eight quarters of play with decisions due at daily 
intervals, except on weekends. The actual schedule is shown in 


Appendix F. 


The schedule determined the length of the console sessions at 
MIT and the time between decisions. A three-hour console session 
was scheduled for each decision period. Each firm had its own 
console during these sessions. In addition, a console was 
available at Endicott House during the experiment. The time 
between decision periods was never less than twenty-four hours 
but the major decision making activity occurred during the 
scheduled console/work sessions. Firms felt that console avail- 
ability and the length of time between decisions were sufficient 


for high quality decision making. 


Assistance was available at all times during the simulation 
exercise as part of the course. The assistance was provided by 
the Game Administrator, Professor David N. Ness, and two assistants, 
Albert Marcotte and Freddy Meurs. Assistance needed by the firms 
usually involved clarification of facts concerning the simulation 
or instruction in the use of the decision support programs. Firms 
requested aid; none was volunteered. On the whole, requested 
assistance was nearly equal between markets and across firms. More 
assistance was required in early periods and decreased exponentially 


during the game. 
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4.2 Conducting the Experiment 


Conducting the experiment was synonomous with running the Game. 
In other words, the gaming exercise was the experiment. The 
experiment, however, was transparent to the decision makers because 
the game was a scheduled event in the Senior Executive Program, 
and no data was overtly collected nor did the researcher consciously 


influence the participants. 
Schedule 


Since the experiment and the game were identical, the game 
schedule given in Appendix F and depicted in Figure 4.7illustrates 


the important events in conducting the experiment. 


First Week Second Week 
Console Session Console Sessions Postgaame 


17 +18 19 


Game manuals distributed 

Introduction to Game (Gg ), to consoles (Gc _ ) 
Scheduled console sessions for quarter number 
Questionnaire distributed 

Interview sessions 

Debriefing sessions 


Figure 4,7 Summary of Game Schedule 


Certain aspects of this schedule are considered here; the 
distribution of game manuals, the introduction to the game, console 


sessions and debriefing sessions. 


The 191-page Game Manuals contained all of the documentation on 
the simulation, the decision support packages and the firm and 
market history for the previous eleven quarters. The Table of 
Contents and a synopsis of each section's contents are given in 


Appendix G . It was distributed five days prior to the first 
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scheduled console session, because this much time was required for 
the decision makers to familiarize themselves with the complex 
material. During the Game, no particular emphasis was placed on 
any of the decision aids. No further introduction was provided 
for any aid other than the material supplied in the game manual. 
Participants were neither encouraged nor discouraged from using 
any of the systems. Use was left entirely to their discretion. 

A session on console use covering all aids was held prior to the 
beginning of the exercise. Since SAS was more complicated, the 
majority of time was devoted to it. The experimenter did not 


bias the experiment towards the use of MDS. 


The introduction to the Game by the Administrator consisted 
of a three-hour lecture followed by a question and answer session. 
Emphasis was placed on the simulation rather than on the decision 
Support packages. During this session, participants were given an 
overview of the Game and performance evaluation procedures, 
The firms knew what goals would be used to evaluate firm performance 
and how achievement was measured. No particular instructions were 


given to participants that might have biased the experiment. 


Eight console sessions, one per period, were scheduled for three 
hours each over a two week period. Thus, decision makers were 
involved in the experiment for at least twenty-four contact hours, in 
the afternoon as noted on the schedule. These sessions can best be 
described as "work sessions" in which consoles and access to the 


decision support programs were available to each team. 


The console sessions became the focus of the game. The mode of 
operation involved returning results of the previous period no later 
than the beginning of the session. Decisions and associated infor- 
mation were collected from the firms at the end of each scheduled 
session. Using this input, the game was "run" that evening and the 
cycle repeated. The debriefing session was conducted after ques- 
tionnaire and interview data was collected. Therefore, data could 
not have been biased by the debriefing. 
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4.2.1 Nature of Experimental Procedures 


The experimental procedures centered on conducting the 
experiment and collecting data which was used as the basis for 
hypotheses tests and evaluation of the validity of the experiment. 
The data collection procedures are characterized by the methods, 
timing and type of data collected. The methods for data 
collection included: 


(1) document collection 

(2) computer tracing 

(3) questionnaires 

(4) interviews 

(5) direct observation 

Table 4.lsummarizes the data collection procedures. Data 

related to the experiment was collected from varied sources by 
different methods and at different times. The times are relative 
to the game; before the game (B), during the game (D), and after 
end of play (E). The methods are document and trace collection (C), 


direct observation (0), questionnaire (Q) and interview (I). 


Table 4.1 


Summary of Data Collection 


Type of Data Time Method 


(1) Firm Goals 

(2) Decisions/Plans 

(3) Firm Reports 

(4) Administrator Reports 

(5) Trace of Console Sessions 
(6) Decision Processes Data 
(7) Working Documents 

(8) Questionnaire 

(9) Post Game Interviews 


Hen moooUYe So 
BHOQDCANAAAN 
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Table 4.1 highlights the unobtrusive nature of data collection 
procedures used in this study. As shown, all documents collected 
during the game were by-products of the exercise. The trace of 
each console session was collected automatically by routines 
internal to the game. Decision process. observations were 
dictated in private by the experimenter at the end of each day's 
session. Collection of working documents, questionnaires and 


interviews took place after the end of the game. 
Document Collection 


Document collection was used to gather needed data that was 
produced in running the game. Since the researcher acted as a 
gaming assistant, he had natural access to all documents and 
handled both the collection of Input Forms and the distribution 
of results for the game. To xerox copies of input documents and 
have the computer print extra copies of results was a simple matter. 


Document collection is summarized in Table 4.2 


Table 4.2 


Document Collection 


Name of Collection 

Document Time/Source 

Data Input Form Quarterly/Firm 

Market Summary Quarterly 

Firm Reports Quarterly/Firm 
Individual Goals Quarter 11 

Firm Goals Quarter 12 
Administrator's Report Quarterly/Firm 

Console Carbon-Trace Console Session/Firm 
Working Documents End of Game/Participant 
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Computer Trace 


The computer trace captured each firm's use of the interactive 
decision aids. Trace routines that were part of the programs 
automatically stored every interaction between the user and 
MDS, FPS, and SAS on tapes. Use was automatically time coded. 


This trace included data for forty-eight console sessions. 


Trace data was similar to that from the console carbon trace. 
Captured on tape, the trace data, therefore, did not require any 
further processing prior to analyses. A sample of this data is 
shown in section 3.2.1. Console carbon paper trace, on the 
other hand, served as a check on the computer trace. In addition, 
the carbon paper trace contained written comments and analysis 


which were not captured by the computer trace. 


Questionnaire 


The extensive questionnaire data was collected after the end of 
the experiment. The questionnaire shown in Appendix H was used 
to obtain data for the purposes shown in the eight areas listed 
in Table 4.3 
Table 4,3 


Questionnaire Contents 


Section Purpose Questions 
I Personal contributions and reactions to game 10 
II Perceived versus actual understanding of game, 
information used for decision making 26 
III Individual and corporate goals, policy and 
objectives 9 
IV Organizational influences 13 
V Perceived usefulness and quality of decision 
support packages 16 
VI Influence of model and information quality on 
results 37 
VII Determination of transfer of knowledge perceived 
similarities and/of differences between markets 17 
VIII Perceived realism and complexity of markets 
awareness of experimentation 23 


pe al Se a ne 
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The questionnaire contained scaled response items, closed and 
open-ended questions. Most of the questions were pretested on 
prior groups of Senior Executives to determine reliability and 
improve format and clarity. Several questions were redundant; 
critical issues and measurements were addressed by multiple questions. 
The questionnaire required two to five hours to complete; the average 
time was three and a quarter hours. Every Senior Executive returned 
a completed questionnaire before the debriefing session. This was 
most gratifying because the game marked the close of formal classes 
and the participants had busy social schedules and graduation 
exercises to attend. Most returned the questionnaire prior to the 
interview. The experimenter checked each questionnaire as it 
arrived and, therefore, was able to clarify any information during 


the interview. 
Interview 


The interviews were conducted with each firm during the week end 
following the completion of game play but prior to the debriefing 
session. The researcher and a helper conducted these interviews 
at Endicott House, the Senior Executive residence at MIT. The 
helper ran the recording equipment and made sure no questions were 
overlooked. All interviews were tape recorded for future reference. 
Tapes are of sufficient quality to permit accurate transcription 


even when a number of people responded simultaneously. 


The interviews were structured around a series of pretested 
questions which were identical for each team. The interviewer 
encouraged clarification of responses to interview question and 
answers to particular questions in the questionnaire. To answer 
all questions required one hour and forty-five minutes to three 
hours; the average was two hours and forty minutes. These fourteen 
hours of tape recorded interviews provide data that were difficult 


to capture in any other way. 
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Direct Observation 


Additional data was obtained by direct observation. Forms 
designed for rapid data entry were used to collect quantitative 
data; therefore, firms were not disturbed by the data collection 
procedures. The data included recording man-hours and resolution 
time for foreign and domestic marketing decisions. Qualitative 
observations were dictated on tape by the experimenter at the 
close of each day. This data served mainly as a check on some 
aspects of questionnaire and interview data and to supplement 


information concerning the dynamic nature of decision making. 


4.2.2 Associated Experimental Conditions 


The physical environment was more than adequate for purposes 
of the experiment. All console sessions were scheduled in a 


large room at the Alfred P. Sloan School of Management at MIT. 


The floor plan and arrangement of facilities is shown in Figure 4.8 


onsole congole congole 


Figure 4.8 Physical Arrangement of Facilities 


The room was large enough to assign a work area to each firm. 
Each area contained its own large conference-type table. The work 
areas defined privacy zones; firms worked within their assigned 
areas and did not violate the confidentiality of the work of other 


teams. Firms used the same tables and consoles each session. 
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The room also contained three wall-mounted blackboards which were 
used for instructional purpose by members of the firms and the 


Game Administrator and his assistants. 


The consoles used in the experiment were IBM 2741's or their 
equivalent. Only hard copy was produced; no CRT consoles were 
used. To capture console use, three-part carboned paper was used 
by each firm; the original was used by the team while the carbons 
were collected by the experimenter. Thus the carbon "trace" 
acted as a backup to the computer trace. In addition to capturing 
all of the console input and output, the carbons contained notes 


and calculations made by teams at the console. 


Posted on the wall next to each console were flow diagrams 
detailing the operation of each of the decision support programs 
and log-in procedures. These handy references served as guides 
to program use, especially in Quarters 12 and 13. The flow 
diagrams remained posted for the duration of the experiment. 


Firms indicated that the diagrams were valuable aids. 


The location of the experiment allowed participants to use 
other resources of the Sloan School (Xerox machine, calculators, 
etc.) and was convenient to dining facilities. Thus, many firms 
made extensive use of the room and facilities at times other 


than scheduled console session. 
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4.3 Implication for Internal and External Validity 


This section evaluates factors related to the design and 
conduct of the experiment which have implications for the validity 
of this work. The most critical factors relate to: 

(1) the similarity of the Foreign and Domestic 
decision environments 
(2) the structure of the experimental design 
(3) the procedures for conducting the experiment 
Each of these areas is discussed and the validity of the research 


design is evaluated using criteria from the literature. 


Validity determines the degree of confidence that can be 
associated with results. Both internal and external validity are 
intimately related to the design and ganauce of any experiment, 
especially in situations where the independent variable is 
manipulated to determine its effects. This is the case here. 
Consequently, those factors within the framework of the experiment 
that can threaten either type of validity are considered in greater 


detail. 


Factors which jeopardize validity have the potential to 
confound results. What can threaten validity is the possibility 
that some other variable is responsible in some degree for the 
observed results. It can do this by reducing, enlarging or being 


entirely responsible for the observed effects. 


4.3.1 Issues Related to Similarity of the Markets 


Most of the methods for achieving high internal validity have 
been called experimental control. The goal, of course, is to 
design experiments which regulate all factors which could confound 
results. The design used in this study has a number of features to 
accomplish this aim. Because the design is intimately related to 
the nature of the foreign and domestic decision environments, the 
similarity of these markets will be discussed prior to discussing 


factors related to the design. 
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Similarity of the Decision Environments 


There are two critical issues related to the similarity 
of the decision environments; actual similarity and perceived 
similarity of the Foreign and Domestic Markets. Actual 
similarity relates to the degree to which both markets are 
structural facsimilies. Perceived similarity, on the other 
hand, relates to the firm's awareness of the underlying 


structure of each market. 


These issues are important because the research design required 
firms to be simultaneously exposed to two similar decision making 
environments, one of which was aided by a Decision Support 
System while the other was not. Therefore, the Foreign and 
Domestic Markets should have high structural similarity. The 
design also required that firms should not recognize basic 
structural similarity and, thus, transfer knowledge gained in 
one market to the other market. The nature and complexity of 
the model which generated the markets allowed both goals to be 


achieved to a very high degree. 


Actual Similarity 


To evaluate the actual similarity of the Foreign and Domestic 
Markets requires an appropriate test. Brodbeck (18 ) suggests 
that in cases where two models are being compared the degree 
of similarity can be judged by a "Turing-type" test. In other 
words, if there were separate models for each market and each 
firm's decisions were comparable, how similar would the behavior 
of both markets be given the same starting conditions? This 
test cannot be applied directly here because the same model 
generates both markets and, in addition,the same observable 


behavior could result from different causes. 


The basic idea behind this test can be used, however. Morris (73) 
asserts that behavior is a function of structure. Everything else 


being equal, structurally identical models should produce similar 
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behavior. Since the markets are simulated, the structure is 
defined by (1) the variables, (2) the relationships among variables 
and (3) the influences of each variable on the total market. 
Therefore, the degree of similarity can best be determined by 
comparing how the model is used to generate the Foreign and 


Domestic Market. 


The criteria for comparing structural identity suggests that 
the markets are highly similar. Since each market is generated 
by the same model, both react to exactly the same twenty 
variables - GNP, a seasonal index, six prices, six promotion 
expenditures and six R&D expenditures. The pattern and values 
for the seasonal index as well as the effects of prices, 
promotion and R&D are identical for both markets. The relation- 
ships among factors are also the same for each market. The 
multipliers for GNP and the breakpoints for curves are slightly 
different, however. Even though breakpoints are different, 
the same percentage change in comparative decision variables 
has identical effect in either market and, thus, is of no 
consequence. The multipliers for GNP cause the Foreign Market 
to grow slightly faster than the larger Domestic Market, all 
else being equal. This effect, however, is completely over- 
shadowed by the action of firms; that is, each competitor's 
price, promotion and R&D strategy has the more pervasive 
influence on each market since the number and nature of their 
relative influence determines, in large measure, the status 


of the market each quarter. 


To summarize, actual similarity is high; the Foreign and 
Domestic Markets can be considered structural facsimiles because 
the same relationships determine each market's behavior. The 
combined actions of the six firms and the seasonal index dominate 
the effects of slight differences in GNP. Consequently, it 
would seem that the inherent behavior of each market creates 


similar decision environments for the firms. 


113 


High structural similarity of the markets means that essentially 
the same decision environments exists for each firm for both the 
experimental and control condition. The conditions are created 
solely by the use of MDS in the Foreign Market while precluding 
its use in the Domestic Market. Consequently, any confounding 
between the nature of the decision environments and the experimental 
and control conditions is reduced to minimum because of the high 
structural similarity of the markets. Differences in decision 
making behavior are, therefore, attributable to the experimental 


variable - the MDS — and not to differences between markets. 


Structural similarity does not imply that each firm encountered 
identical decision situations each quarter. Circumstances are 
unique but they are created by the firms themselves and not by 
the structure of the markets. The situation facing each firm is 
primarily a function of its past decisions and the past and future 
decisions of its five competitors. Since no firm has exactly the 
same decision history nor the same competition, individual firms 
are faced with unique circumstances each quarter. These circum- 
stances do not influence the experimental and control conditions, 


however, because the analytical procedures recognize all differences. 


Perceived Similarity 


Forrester's (42 ) work alludes to the difficulty in detecting 
the structure of dynamic, non-linear models from their observable 
behavior. This difficulty was clearly evidenced in the game 
because dynamic interactions tended to obscure and confound one 
another. As a result, firms believed that the markets were 
distinctive, not knowing that any differerences were, in fact, 
determined mainly by their own interactions. Moreover, participants 
never suspected at any point in the game that both markets were 


essentially identical. 
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These conclusions are clear from the questionnaire and 
interview data. Both contained a number of items related to 
perceived similarity of the markets and the possibility of 


knowledge transfers between markets. 


(Questionnaire, item  ) 


In your mind, how similar are the Foreign and Domestic Markets? 


Responses 


15 


10 


Highly Moderately Similar Moderately Highly 
Similar Similar Dissimilar Dissimilar 


Figure 4.9 Perceived similarity of markets 


(Questionnaire, item ) 
9 


If you think they are similar, why do you believe this to be the case? 


“They were slightly similar. Some of the same kinds of 
variables like GNP and Price effected both, but other 
variables like Advertising and R&D didn't have the same 
influence at all" 

"Not too similar - they certainly weren't the same market. 
The manual says they were influenced by the same variable - 
but I don't believe it" 

"The only thing you could say about the similarity is that 
some things seemed to behave the same way some of the time" 
"T don't think they were similar at all - in fact, they were 
completely different" 
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To determine knowledge transfer between markets, the series 


of questions shown below were included in the questionnaire. 


(Questionnaire, item ) 


Which market do you feel you understand better, the Foreign Market 


or the Domestic Market? 


Responses 


15 


-10 


Foreign Market Domestic Market 


Figure4.10 Comparison of Understanding 


(Questionnaire, item ) 


How useful was the knowledge gained in the Foreign Market for 
understanding the Domestic Market? | 


Responses - 


Extremely Highly Fairly Not 
Useful Useful Useful Useful 


Figure 4.11 Usefulness of knowledge, Foreign to Domestic 
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(Questionnaire, item  )}. 


How useful was the knowledge gained in the Domestic Market for 
understanding the Foreign Market? . _ | 


Responses 


1 
Extremely Highly Useful Fairly Not 
Useful Useful Useful Useful 


Figure4.12 Usefulness of knowledge, Domestic to Foreign 


(Questionnaire, item  ) 
How would you rate the transfer of knowledge between the markets? 


Responses 
4 15 
10 
5 
Very High - Medium Low Very 
High | Low 


Figure4.13 Knowledge transfer between markets 


This data clearly shows that the similarity of the markets 
was not perceived nor was knowledge gained in one market particularly 


useful for competing in the other market. 
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4.3.2 Issues Related to Design of Experiment 

Alternative designs are well documented in the literature 
(99 ) (82) (97). They are classified into three categories by 
Campbell and Stanley (20) based upon the researcher's ability 
to control factors influencing validity. Pre-experimental and 
quasi—experimental designs are characterized by lack of complete 
control. True-experimental designs, on the other hand, are usually 
characterized by random assignment of subjects to experimental 
and control groups, full control over experimental stimuli and 
parallel exposure and observation of both groups to determine the 
effects of the experimental stimuli. True-experimental design 
has higher inherent validity than do the other two types and for 


this reason are preferred if conditions permit their use. 


The design used in this study is a true-experimental design. 
Certain features noted at the bottom of Figure4.14 further increase 
the basic design's inherent validity. Rather than assign "subjects" 
by the usual randomizing techniques (which would have permitted 
using only three firms for each market) the design used related 
samples instead. Siegel (02) notes the advantages: 


"In comparisons of two groups, sometimes 
significant differences are observed which 
are not the results of the treatment... One 
way to overcome the difficulty imposed by 
extraneous differences between groups is to 
use two related or matched samples." 

Matching can be achieved by using each subject as his own 
control or by pairing subjects and then assigning the two members 
of each pair to the two experimental conditions. When a subject 
"serves as his own control", he is usually exposed to both treat-— 
ments at different times. When the pairing method is used, each 


pair must be as much alike as possible with respect to extraneous 


variables which might influence experimental results. 
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Even though matched-pairs and own-control designs are superior 
in themselves, there is potential for improvement. Their main 


advantages and disadvantages are shown in Figure 4.15. 


Matched-Pairs Own-Control 
Design Design 
Simultaneous Perfect 
exposure to Matching 
Advantage experimental 
and control 
condition 
Sequential 
exposure to 
Disadvantage Poor matching experimental 
and control 
condition 


Figure 4.15 Comparison of Participant Assignment Methods 


The experimental design used here combines the advantages of 
matched-pair and own-control designs while overcoming the 
disadvantages of both. That is, perfect-matching was achieved 
because each firm acted as its own control and, in addition, each 
firm was simultaneously exposed to the experimental and control 
condition. No more precise matching is possible than that achieved 
by identity. Thus, differences related to the subjects are 
minimized, while simultaneous exposure to conditions precludes the 
possibility that extraneous variables might influence experimental 
results. Besides simultaneous exposure to the experimental and 
control condition, the design permitted parallel observation of 
the decision making behavior of each group in both states for 
eight replications of the experiment. Parallel observation 
increases confidence since no time-dependent effects are recorded 


by the data collection methods. 


Several other features deserve consideration. All of the 
necessary arrangements were established prior to the beginning of 
the experiment. No testing or data collection of which the firms 


were aware took place before or during the experiment. All data 
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collected during the game was furnished as a by-product of running 
the game. Questionnaire and interview data was collected at the 
end of the exercise. Moreover, the Senior Executives who partici- 
pated in the experiment were totally unaware of the nature of the 
research; the experiment was transparent. From their point of 
view, they were involved in a scheduled management gaming exercise. 
From the researcher's point of view, the game was a laboratory in 
decision making behavior. All participants in the experiment had 
been exposed to identical programs at MIT, thus, all training was 
equivalent for seven weeks preceding the experiment. In addition, 
all were present for the game introduction and no one dropped out 
of the experiment once it was started. None had prior experience 
with Decision Support Systems. The researcher acted as a gaming 
assistant during the game. Having had prior gaming experience 

and being fully aware of the nature of the experiment, he 
dilligently avoided any interaction which had the potential for 


influencing results. 


At this point the internal validity of the experimental design 
can be technically assessed. This requires identifying factors 
which threaten validity by confounding the effects of the experi- 
mental stimuli and thus serve as rival hypotheses for explaining 
the results. Campbell and Stanley ( 20) summarize the literature 
by identifying the impact of: 

(1) Selection ~ biasing from subject assignments 
(2) History - effects of intervening events 

(3) Maturation - time dependent effects 

(4) Testing - effects of multiple testing 

(5) Instrumentation - effects of measurements 
(6) Mortality - differential loss of subjects 
(7) Interaction - interaction of factors 

The experimental design used in this research has high internal 
validity when measured in terms of these factors. The effects of 
selection are not relevant because each firm served as its own 
control. There were no selection biases; matching by identity is 


exact. The effects of history should be recognized. Console 
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problems influenced one team for a quarter but, on the whole, 
events were applied equally to both experimental and control 
conditions. Maturation or the effects of time dependent psycholog- 
ical processes (97 ) would effect both conditions equally because 
of simultaneous exposure. The effects of testing and instrumen- 
tation were nil; none was done during the experiment. Mortality 

or the loss of team members did not occur. Interaction between 
factors would seem negligible. Because the game was a scheduled 
part of the Senior Executive Program and the experimenter's 

role was non-reactive, the well known "experimenter" and 


"Hawthorne" effects are not plausible hypotheses either. 


To summarize, the design controls all of these factors and, 
therefore, has high internal validity. Consequently, observed 
results should be associated with the experimental variable 
rather than with extraneous factors which might have had the 


potential for influencing results. 


4.3.3 Issues Related to Conduct ‘of the Experiment 


The use of teams in conducting the experiment rather than 
individuals was dictated by the situation rather than the design. 
Team management, however, does reflect the trend toward organiza- 
tional control by professional managers rather than the lone 
entrepreneur working in isolation. Farris (41 ) notes this trend: 


"Executive decision making in organizations... 
is seldom done by individual members of the 
organization acting alone. People work 
together in project teams or task forces, 
coordinate their efforts with broader purposes 
of the organization, and exchange stimulation 
and support with their colleagues." 


Team composition is not of compelling relevance because the 
experiment used each firm as its own control. Results of the 
experiment are based on comparison of decision making effectiveness 
between markets rather than between teams. Therefore, imbalance 
between teams, if it existed,is not a significant factor in 
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The use of teams has implications for external validity, 
however. The results may extend to management situations 
involving teams or groups of decision makers but may not apply 
to situations in which decision makers work in isolation. Although 
this is a limitation of the study, it may not greatly restrict 


generalizability. 


Campbell and Stanley (20 ) identified subject-related factors 
that interact with the experimental stimuli (X) and, thus, threaten 
external validity. They are: 

(1) interaction of selection and (X) 

(2) reactive arrangements and (X) 

(3) other interaction with (X) 
Each of these factors centers on the relationship between the 
teams and the Decision Support System or other experimental 
arrangements and attempt to uncover effects that are unique to 
specific populations, conditions or environments and thus reduce 
generalizability. For this study, there are no apparent inter- 
actions that are directly related to teams and the MDS because 
all firms had little experience with Decision Systems prior to 


the experiment. 


122 


4.4 Chapter Summary 


This chapter completely documents the design and conduct of the 
experiment and discusses implications for internal and external validity. 
Design goals are reviewed and translated to specific structural features 
of the experiment. Associated design considerations related to the firms 
and the experimental situation detail how the researcher worked within 
an established gaming exercise. All procedures and conditions for con- 
ducting the experiment are described. 

Extensive data collection methods reveal the wealth of supplemental 
information collected during and after the actual experiment. 

Questionnaire data is used to explore all of the technical issues 
related to internal and external validity. Each technical factor gleaned 
from a thorough review of the literature which had the potential for 
jeopardizing validity is analyzed and considered for this research. The 
conelusion is clear; this experimental design neatly controls all of these 
factors and, therefore, has extremely high internal and external validity. 

Compared to traditional designs found in the literature, this 
experimental design is unique. The design resulted from an overt 
consideration of the goals of good research. This chapter clearly shows 
that the experimental design used in this study either meets or exceeds 
all of these goals, and, therefore, high internal and external validity 


is assured. 
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CHAPTER 5 


DESIGN OF MEASURES AND METHOD OF ANALYSIS 


The measures used to determine decision quality of DSS-aided 
and non-aided decision making are an important part of the 
research design. The quality of measurement depends upon the 
method of analysis. The value of the analysis must be determined 
in light of the needs of the study. The goal was to determine 
the relative, not absolute, quality levels for decision quality 
for every firm for each quarter. The quality of the method 


determines the procedural validity for the study. 


This chapter describes the procedures, presents the table 
of quality indices which it produced and discusses issues related 
to procedural validity. The organization and contents of the 


chapter are shown in Figure 5.1. 


5.1 Measures of Performance 


5.1.1 Objectives 
5.1.2 Measures 


5.2 Method of Analysis 


5.2.1 Procedure for Measuring Decision Quality 
5.2.2 Quality Indices 


5.3 Implications for Procedural Validity 


5.3.1 Issues Related to Measures 
5.3.2 Issues Related to Procedures 


5.4 Chapter Summary 
Figure 5.1 Contents of Chapter 5 


The first section describes the objectives of measurement and 
translates them into appropriate measures. The second section 
describes the procedure for measuring decision quality while the 
third section discusses the implications of the measures and the 


procedure for validity. 
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5.1 Measuring Performance 


The measures and methods used to analyze decision performance 
are critical parts of the research design. They deserve special 
consideration because they determine the quality of the results. 

The measurement of aided and non-aided decision performance requires 
measuring effectiveness and efficiency. Effectiveness is related 
to quality, consistency and rate of improvement in decision making 


while efficiency is related to resource use and resolution time. 


The measurement of effectiveness requires finding a 
standard of comparison. This standard is the "potential" level of 
decision quality. The analytical procedures described here were 
used to find the decision quality levels for each firm in both the 
Foreign and Domestic Market over eight quarters. The measurement 
of efficiency requires data on man-hours and the resolution time 
for foreign and domestic marketing decisions. This data was 
collected in a straightforward way from the experiment. Since the 
measurement of decision effectiveness is more complicated than the 
measurement of efficiency, this chapter will focus on the methods 


of analysis and issues involved in measuring effectiveness. 


3.1.1 Objectives 


The primary objective in measuring decision effectiveness 
was to develop reliable measures which would recognize the unique 
situations facing each firm in both markets and thus provide a 
basis of comparison between firms in each quarter. Since conditions 
encountered by each firm are unique, measures must account for 
subtleties in each situation. These measures of effectiveness are 
based on the decision quality of each firm. To measure decision 
quality requires developing an index which reflects the relative 


value of actual decisions to the firm. 


To evaluate the quality of any activity requires the 
comparison of actual performance with some standard which would 
provide an indication of the worth of the actual activity relative 
to this standard. Thus, it is a measure of performance for the 
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activity. Furthermore, to compare the quality of two or more 
activities requires the comparison of both against a comparable 
standard so that the relative quality of each has significance 


and can be evaluated. 


In the game, a number of different comparisons of actual 
to standard could have been made in attempting to evaluate decision 
quality. The most meaningful, however, is the ratio of profits 
earned as a result of actual decisions to those which would have 
been earned had the firm's decisions been optimal or near optimal. 
The ratio of actual profits to potential profits is a meaningful 


index of decision making quality. 


This measure of comparison was adopted primarily because 
it overcame any differences between the Domestic and Foreign Market, 
even though the markets are structurally identical. These 
differences result from two sources: 
(1) the levels and rates of growth of GNP 


(2) the market conditions created by the 
interactions of each firm's decisions. 


The former is not under the control of the firms, but is determined 
by the Game Administrator. The latter is created by the inter- 
actions of the firms. Thus, each firm faces a unique set of 
conditions each decision quarter. These conditions are a function 
of the past decisions of both the firm and its competitors and the 


future decisions of competition. 


The uniqueness creates problems in assessing the effect- 
iveness of decision making and in comparing the quality between 
firms. For example, the profit potential and actual profit for 
each firm may vary from quarter to quarter and may vary among 
firms within the same quarter, thus making actual profit a poor 


measure of decision quality. 
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The only way to overcome these problems is to determine the 
profit potential for each firm in each quarter and use this as a 
base for comparing actual profits. In this way differences between 


markets may be reconciled. 


The measurement objectives can be realized by basing results 


on ratios of: 


Actual Profits 
Potential Profits 


5.1.2 Measures 


The dimensions of decision effectiveness and my methods of 


measurement are summarized in Figure 5.2 


Figure 5.2 Measures of Decision Effectiveness 


Dimensions Measures 


Decision quality level The ratio of actual profits/potential 
profits. This index compares the 
profits resulting from the actual 
decisions to the potential profits 
which would have resulted had decisions 
more closely attained goals. 


Consistency of Decision The variance of the indices of the 

Quality decision quality level. The variance 
measures the fluctuation in decision 
quality. 

Improvement in Decision The net rate of change of the indices 

Quality of decision quality. The net rate of 


change is an indication of the rate 
improvement. 
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2.2 Method of Analysis 


To determine the quality indices requires finding the ratio 
of actual profits to potential profits. Therefore, an expression 
for profitability is needed. Marketing profitability for a quarter 
or, more precisely, the contribution to the firm's profits from 


the marketing function is given by the expression. 


= os = am * = i 
P ( Pg m)* 8, - cy * a, ~ Co *¥ Cy *i 


q q 


P, is marketing profitability for quarter 
where Sq is sales for quarter 

Pq is price for quarter 

mq is Marketing's cost of product 
is advertising expenditure for quarter 
Tq is R&D expenditure for quarter 
q is inventory carried into period 
cj is an advertising allocation multiplier 
cg is an R&D allocation multiplier 
c3 is the inventory carrying cost 


This expression accounts for all decision variables under the 
control of marketing in the game. It is also a valid representation 
for profitability in both the. Foreign and Domestic Markets given 


appropriate values for the Ci parameters. 


Finding actual profitability for any quarter requires 
substituting the actual values for the decision variables 
(p, a, r, i) and the resulting sales into the expression and 


is no problem. 


Determining potential profitability for any quarter, on the 
other hand, requires finding values of the decision variables 


that maximize P,. The simplicity of the task is deceptive. 


There are four decision variables: price, advertising, 
R&D, and inventory set by each firm each quarter. Given the 
actions of the five competitors with respect to each of the 
decisions, finding an optimal set of decisions for the firm is 
a formidable search problem in a non-linear space, especially 
when longer decision horizons are adopted. The space is essentially 
infinite. Analytic solution techniques for non-linear surfaces 


of this nature are not available. 
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The problem becomes even more difficult when the time horizon 
is lengthened. It is related to the sequential interdependencies 
between quarters. The feasible set of decisions for price, 
advertising and R&D in any one quarter is based upon those of 
last quarter and should include consideration of expected consequences 


in future quarters. 


Clearly, the analysis must account for dynamic interdependencies; 
that is, sets of decisions must be optimal with respect to each over 
an appropriate decision horizon. A four quarter decision horizon 
was chosen in the analysis. This horizon matches the planning 
horizon each team used during the game. Moreover, sensitivity 
studies indicated it was the more appropriate horizon given the 


prevalent seasonal sales patterns in the market. 


The objective function for finding firm profitability is to 


maximize ath q+4 


a CR me SE a ge a 
over a rolling four quarter decision horizon. The procedures are 


explained in later sections. 


As noted above, complete search across all variables in 
the static (one quarter decision horizon) case is formidable. 
For the dynamic case the task is clearly not feasible. Even 
partial search is extremely difficult, costly and time consuming. 
There are three basic ways to reduce search: (1) reduce the 
number of variables over which search takes place, (2) reduce 
the increment size used for each variable, (3) learn what 
solutions are dominant and thus be able to disregard significant 


parts of the space. 


It became clear that price was the most sensitive decision 
variable because its effect was not smoothed and it was the main 
determinant of potential contribution. Because of the smoothing 
effects on advertising and R&D, the results of changes in these 


variables effects sales much less than does a small change 
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in price in the quarter in which all decisions are made. For this 
reason advertising and R&D expenditures were accepted as given for 
all firms and all search was confined to the price/inventory 
variables. This decision substantially reduced the required 
search to a manageable level and produced reasonable estimates for 


potential profitability. 
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TET 


6.20 895 
6.70 964 
6.80 1160 
9.00 4435 
9.00 3549 
9.00 3172 
9.00 2880 
7.50 2834 


Domestic Market Non-Aided Decision Making 


6.40 906 
6.25 1022 
6.25 1265 


9.50 
9.50 
9.50 1059 
9.60 1461 


6.40 906 

6.25 1022 

6.25 1265 

5.80 1758 
.00 


20 
90 1059 
-60 1461 


6.40 906 
6.25 1022 
6.25 1265 
5.80 1758 
7.00 1550 
7.20 1548 
7.20 1597 
7.20 2519 


8.50 420 
8.40 834 
8.50 885 
9.50 1305 
9.50 987 
9.50 987 
9.50 1059 
9.60 1461 


Foreign Market Aided Decision Making 


Figure 5.3 Analysis Sheet 


6.40 906 
6.25 1022 
6.25 1265 
5.80 1758 
7.00 1550 

1548 

1422 


6.25 1022 
6.25 1265 
5.80 1758 
7.00 1550 
7.20 1548 
7.20 1422 
6.20 2099 


5.2.1 Procedures for Measuring Decision Quality 


The easiest way to understand all of the procedures used to 
find the dynamic optimal decisions and their associated potential 
contributions is in reference to the analysis sheet shown in 
Figure 5.3. The upper half is the analysis for the Domestic Market 
and the lower half is the analysis for the Foreign Market. 


Above the diagonal step-lines and at the extreme right are 
the actual decisions and contributions earned each quarter. 
Below this line are the optimal sequences of decisions given the 
"state of the world" as it appeared in the quarter in which the 


optimal sequence starts. 


For example, in the Domestic Market in the column labelled 
Q15, the optimal pricing policy for quarters 15 through 19 is 
shown below the diagonal. This is the optimal policy given that 
the firm had charged $6.40, $6.25 and $6.25 in quarters 12, 13 and 
14 respectively. By looking to the right of the $9.02 optimal 
price on the line for quarter 15 in the column labelled Q16, it 
can be seen that the firm actually charged $5.80 and in the 
process stocked out. Note that the optimal sequence of prices 
for quarter 16t 919 given that the firm actually charged $5.80 
in quarter 15 is given below the diagonal in column labelled Q16 


is different from the sequence with an optimal Q15 decision. 


To find the sequence of dynamically optimal decisions and 
the corresponding potential contributions for each quarter the 


objective function 
+4 +4 x ‘ A e405 
5 Pgs > ((Pg - cf eames aq — ¢2 tq -— ¢3 1 
qd q 


was used to guide sequential search through the price/inventory 
space until the sequence of prices that maximized the objective 
function was found. The procedure was repeated beginning in 
each subsequent quarter thus producing the sequences of optimal 
decisions below the diagonal line for both markets. In moving 


from quarter to quarter the firm's actual decisions which 
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determine the "state of the world" were taken as the starting point 


in finding the new optimal sequence. 


The potential contribution that could have been earned by the 
firm each quarter is shown below the box under the appropriate 
quarter. It is the sum of the actual contributions and the optimal 
contributions shown above in the column. It represents the 
potential contribution which was lost due to non-optimal (actual) 
decision making. For example, in Q15 the difference in the two 
sequences is the difference between the optimal price of $9.02 in 
quarter 15 and the actual price of $5.80 and the effects of this 
mistake in subsequent quarters. Because $9.02 was not changed in 
Q15, the firm's inventory was sold out at the lower actual price 
of $5.80. In addition, the firm lost the opportunity to sell at 
a higher price of $9.60 in the subsequent quarters which accounts 
for the shift to $7.49 for the optimal sequence once the decision 


to set the actual price at $5.80 was made. 


The analysis is identical for both markets and proceeds from 
quarter to quarter finding the optimal sequence of prices for the 
remaining quarters and the associated potential contributions. 
Subtraction of the summed contribution determines the dollar 
loss due to non-optimal (actual) decision making for any quarter. 
The ratio of potential to actual contribution are the quality 


indices reported in Table ‘ 


In summary, the dynamic analysis determines quite accurately 
the loss due to non-optimal decision making for each quarter by 
considering the future impact of present decisions. All things 
considered, this analysis provides reasonable estimates for 


determining quality indices. 
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un 


e2ed Quality Indices 


The quality indices for every firm for each quarter were 


determined by the procedure. The results are shown in Table 5.2. 


TABLE 5.2 
Quality Indices 


Quality Indices for Non-Aided Decision Making 


Domestic Market - Control Condition 


Quarter Firm 1 Firm 2 Firm 3 Firm 4 Firm 5 Firm 6 


12 0.985 0.974 0.989 0.981 0.988 0.970 
13 0.989 0.961 0.997 0.992 0.976 0.988 
14 0.925 0.890 0.918 0.919 0.936 0.956 
15 0.580 0.617 0.603 0.623 0.613 0.521 
16 0.855 0.807 0.971 0.817 0.820 0.870 
17 . 0.937 0.932 0.947 0.950 0.912 0.940 
18 0.981 0.939 0.853 0.943 0.958 0.872 
19 0.935 1.000 0.732 0.680 1.000 0.792 


Quality Indices for Aided Decision Making 
Foreign Market - Experimental Condition 


Quarter Firm 1 Firm 2 Firm 3 Firm 4 # £Firm5_ Firm 6 
12 0.748 0.742 0.706 0.726 0.759 0.769 
13 0.791 0.819 0.795 0,809 0.798 0.761 
14 0.937 0.954 0.970 0.945 0.943 0.975 
15 0.876 0.920 0.995 0.981 0.971 0.643 
16 0,988 0.850 1.000 1.000 0.959 0.978 
17 0.994 0.992 1.000 1.000 0.981 1.000 
18 0.993 0.961 1.000 1.000 0.986 1.000 

i419 0.978 0.990 1.000 1.000 0.910 1.000 


134 


5.3 Implications for Procedural Validity 


This section discusses the issues related to procedural 
validity. The purpose here is to explore factors which have 
the potential to influence the quality of the measurements 
and, therefore, jeopardize procedural validity. Procedural 
validity concerns the quality of the measures and procedures 
used in the study. Measures, in this case, are quality indices 
and relate to the analytical procedures. These procedures 
determine the reliability of the measurements. This section 
is divided into two parts; (1) issues related to measures, 
and (2) issues related to procedures. Issues related to 
methods are discussed. The relevancy and reliability of 
the measures is discussed before the reasonableness of the 


procedure for finding quality indices is evaluated. 
5.3.1 Issues Related to Measures 


There is consensus in the literature, (20),(47), (87) 
that validity requires measures be relevant and reliable. 


These criteria are used to structure the following discussion. 


Relevance 


Measures must be pertinent to the experiment. For this 
experiment, this implies that decision quality should be 
measured in terms directly related to the goals of each firm. 
Moreover, the measures should reflect how decision making 
effectiveness might be measured in the real world. It seems 
reasonable, therefore, to measure the quality of decision 
making by its effects on the profitability of the firm. This 
recognizes the interrelationship of marketing decision and 
profits and is a reflection of the goals set by the firms in 


the game. 


Table5.2 shows the importance of each goal to the firm. 
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TABLE 5.2 


WEIGHTING OF FIRM GOALS 


Ga i ee 


Market Change in Earning/ Return on Market 
Share Net Worth Share Equity Value of Firm 
(1) (2) (3) (4) (5) 
Firm 1 1 3 5 4 2 
Firm 2 3 1 2 4 5 
Firm 3 2 1 5 3 4 
Firm 4 1 2 4 5 3 
Firm 5 1 2 5 3 4 
Firm 6 1 2 3 4 5 


The table shows that most firms emphasized profit related 
goals. The firms realized that goals 2 to 5 were directly 
related to profitability, because they knew how goal attainment 


was evaluated by the Game Administrator. 


Firms were required to weight each goal on the Firm Goal 
Form shown in Appendix H. These goal weights were used to evaluate 
each firm's performance in the game. Actual rank in the Game with 
respect to these goals was multiplied by goal weights. These 
products were summed for each firm to create an index of performance 
Total points represented by the index determines relative rank 


among the six firms. 


The important question is how these goals influenced firms. 
In other words, to what extent did profit goals guide decision 
making strategy of each firm? Evidence is shown in Figures 5.4 
to 5.6 . Goals clearly served as objectives for the firms as 


evidenced by the questionnaire and interview data shown below. 


136 


(Questionnaire, item ) | 
roals which you submitted to the game administrator 


Did the firm's 
guide your decision making? 
Responses 


15 


10 


Moderately Some of Very 
the time little 


Always Most of 
the time 


Figure 5.4 Influence of goals on decisions 


(Questionnaire, item ) 
Was the goal conflict among team members, if it existed, detrimental 
to your firm? Yes or No _? it interfere w attainmen 
of firm goals? Yes or No _? 
‘Responses Responses 
15 15 
10 
5 5 


Interfere Not 
Interfere 


Detrimental Not 
Detrimental 
Figure 5.5 Influence of goal conflict 
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Interview data confirmed these responses to the questionnaire. 
Some conflict was considered normal but not detrimental to the firms 
nor did it interfere with goal attainment. The concensus was that 
any goal conflicts reflected healthy differences of opinion much 


like what the decision makers experienced in the real world. 


To this point, it has been shown that (1) goals relate 
directly to profitability and (2) goals guided the firm's decision 
making and (3) any conflict which existed did not have a detrimental 


effect or interfere in any substantial way with goal attainment. 


Two important questions remain: (1) did firms attempt to 
maximize marketing profits and, if it is clear that they did, 
(2) did firms attempt to maximize profits on a quarterly basis 
or over some longer decision horizon? Figures and 
provide data related to these questions. 

(Questionnaire, item ) 
If you had perfect knowledge and information, would your primary 


oal have been profit maximization: Yes or No 2 If yes 
would the primary approach have been to maximize marketing 


profits: Yes or No . If no, what would you have done? 


—— Responses Responses - 


fe) Yes No Yes 


Profit Maximization Marketing Profits 


Figure 5.6 Primary goals 


138 


Three of the five "no" responses to the last question 


commented on their reasons. 


“the marketing decisions were certainly important 
but the capacity decisions are more important - 
we missed the timing on expansion." 

"I believe the production decisions were more 
important - maybe because my background is in 
this area - what I wanted to do was run the 
plant at maximum efficiency." 

"The financial decisions were my main concern. 

I did little in the marketing area." 


(Questionnaire, item ) 
Over what number of quarters did you realistically try to maximize 


the results of your marketing decisions? 


Responses 
15 15 
10 10 
5 5 
0 1 2 3 4 5 6 7 8 


Figure 5.7 Marketing Decision Horizons 


All of this data suggests that the primary goal of the firm 
was to maximize marketing profits, if possible, over a decision 
horizon that averaged four quarters since the responses to the 


last question were evenly distributed among firms. 
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Reliability 

The reliability of measurement means that values found for 
a certain measure recurs in repeated measurements under the same 
circumstances. Kerlinger (59) suggests that a measurement is 
highly reliable if replication produces the same or similar 
results. Reliability of measures in the game is related to actual 
results and to the analytical procedures used to derive the 
standards of comparison. Therefore, the reliability of the 
measures has two components which must be considered - the quality 


of actual results and the quality of potential results. 


The quality or accuracy of actual results is not open to 
question. The Game had been used previously and the accuracy of 
reported results verified on a number of occasions. In addition, 
before the game was run, all input data was checked by the 
Game Administrator and his assistants, as was the output returned 
to the firms. The researcher also reran the entire eight quarters 
of play for the experiment on a different machine (larger word 
size) and computed various critical results by hand. In all 


these tests no error in actual results was detected. 


The quality of potential results rests on the analytical 
procedures. Due to the complexity and the number of relevant 
issues, the procedures are discussed in detail in the next 
section, which demonstrates that quality of potential results is 
high, judged by any reasonable set of criteria. Moreover, these 
results are objective and quite accurate, as repeated checks 
have indicated. Therefore, measurements of quality indices can 


be considered highly reliable. 


The reliability of measures also relates to the level of 
measurement attained. Siegel (102) identifies four levels of 
measurement: nominal, ordinal, interval and ratio. Each is 
related to the appropriate choice of the statistical test used 
in the study. An interval level of measurement for the quality 
indices is obtained by the analytical procedures. 
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5.3.2 Issues Related to Analytical Procedures 


The analytical procedures are the basis of measurement. 
They provide the standards of potential profitability for each 
firm each quarter which are used as the denominators of the 
quality indices. Parsons (82) and Mester and Rabideau (67) 
have identified three criteria associated with analytical 
procedures that are particularly important in man/machine 
research: applicability, reasonableness and reliability of the 


procedures. These criteria are used to structure this discussion. 


Applicability 

Applicability relates to the appropriateness of a specific 
technique for achieving useful results. The usefulness of 
results must be judged in light of the needs of the study. The 
research required the determination of the potential profits each 
firm could have realized in each quarter in the Foreign and 
Domestic Market. Potential profits were the denominators of the 
quality indices and, therefore, required sufficient accuracy to 


allow meaningful comparisons between firms. 


To determine potential profits requires an appropriate 
analytical procedure The choice centers on the tradeoff between 
the power of the technique and the realism it could handle. 
Certainly, optimizing models are preferred if their use is 
possible. The requiraments for using optimizing models are 
recognized by Emery (40 ): 


"It must be possible to duplicate the real 
world in mathematical form with sufficient 
accuracy that results from the model to 
make sense and there must be available a 
computationally feasible procedure for 
finding optimal solutions." 


The use of a formal optimizing model was not feasible for 
finding profit potentials. To have done so would have required 


many compromises and, therefore, results would have been unrealistic. 
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The problem involved finding a series of interdependent integer 
decisions for a game world characterized by discontinuities in 
non-linear relationship, high uncertainty with respect to various 
estimates and essentially an infinite number of feasible solutions. 
Several formal techniques were reviewed: linear and non-linear 
programming, integer programming and dynamic programming. These 
efforts showed that to reduce the problem to a representation for 
which optimizing techniques were feasible required compromises 
which would have made the results rather meaningless. It was 
clear that accurate representation of the situations facing each 


firm was mandatory for obtaining useful results. 


Fortunately, the problem was amenable to a search procedure 
without having to oversimplify the unique situation facing each 
firm in the complex decision environment. Search occurred over 
the surface representing the unique conditions created by the 
firm and its competitors. A measure of effectiveness that 
accounted for all of the firms' decision variables was used to 
guide the search. The procedure provided good estimates of the 
potential profits that firms could have achieved had their 
decisions been optimal or near-optimal for the complex situations 


facing firms each quarter. 
Reasonableness 


The reasonableness of the estimates for potential profits 
relate to the measure of effectiveness, the procedures and the 
problem representation. As noted, the procedures did not require 
simplifying the problem but certain features of the analyses 


deserve further consideration. 


Og mene he effectiveness used to guide search was: 


. Pa ac ~ m) *Sq -cy * ag — C2 * Tq - 63 * ig) 


This eprestiod a used to evaluate actual and potential profits 
for both the Foreign and Domestic Markets by searching over the 


price-inventory surface assuming the advertising and research 
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and development expenditures were given, 


The price-inventory space was searched because price was 
the primary determinant of sales in each quarter. Its effects 
were immediate rather than smoothed, as were the advertising and 
R&D effects. Moreover, advertising and R&D decisions were 
established by policy and thus tended to change less frequently 


than price. 


Firms identified the pricing decision as the critical 
decision in marketing strategy each quarter and noted in the 
interview that advertising and R&D decisions were essentially 
policy based decisions which tended to remain stable for longer 


periods of time. 


Questionnaire, item  ) 


What was the most important marketing decision your firm made 
each quarter? 


Responses 


Price Advertising R&D 


Figure 5. 8 Perceived importance of marketing decisions 
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Even more important, to have searched advertising and R&D 
in addition to the price and inventory variables would have made 
the search unmanageable. A number of sensitivity studies were 
carried out early in the research which indicated that profit 
potentials were not significantly affected by confining the 
search to only the two most critical variables. To use more 
variables would have increased the search time substantially 


without a concomitant effect on results. 


The measure of effectiveness is based on profitability. 
Four out of five firm goals directly related to profitability 
and the weights firms placed on these goals indicated that 
achieving high profits was the route that all firms adopted in 
attaining their goals. Moreover, firms attempted to set prices 
so that profits would be"maximized" in the longer run rather than 
on a quarterly basis. Therefore, using a measure of effectiveness 


based on profitability is relevant. 


The measure of effectiveness was used to find price- 
inventory decisions which achieved the highest potential profits 
over a four quarter decision horizon. Firms implicitly adopted 
a similar decision horizon because substantial seasonal peaks 
occurred every four quarters. The firms stated in the questionnaire 


that their goal was to look four quarters ahead, on the average. 
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(Questionnaire, item ) 


How many quarters ahead do you feel that your firm should 
have looked in evaluating the effets of each quarter's decisions? 


Responses 


0, 2 4 6 8 
Periods Ahead 


Figure 5, 9 Decision Horizons 


The console trace and responses in the interview confirm 
that the actual decision horizons adopted were shorter in some 
cases and in addition, decision horizons in the Foreign Market 


were greater than their counterparts in the Domestic Market. 


The effects of different decision horizons were studied 
by the researcher. These studies were based on one, two, three, 
four, five, six and eight quarter decision horizons. Decision 
horizons greater than four quarters clearly required excessive 


computation and unwarranted assumptions for ending conditions. 


The one quarter decision horizon was helpful in that it 
provided a static analysis. Two and three quarter horizons 
were not long enough to recognize the seasonality in the markets. 
Therefore, a four quarter decision horizon seems the better 
choice given the seasonality of the environment, the actions of 
the firms and the increased search required if longer horizons 


were adopted. 
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The experiment formally ended in quarter 19; actual data 
beyond this point is not available. To maintain a consistent 
four quarter decision horizon, firms were asked during the 
interview to develop a plan for three succeeding quarters. 
Results were recorded and used as the basis of estimating effects 
in the last quarters. Prior work had shown that this alternative 
was preferable to using shorter planning horizons for attributing 
potential profitability to quarters 17, 18, and 19. This procedure 
may understate potential profits and thus provides conservative 


estimates of quality indices for the later quarters. 


The effects of previous decision in any quarter determines 
what is feasible in the next quarter. Therefore, prior decisions 
determine the "state of the world" at the beginning of the decision 
sequence and are taken as a given for purposes of analysis. All 


firms had to do this in the game. 


This assumption is needed because past decisions effect 
future results in the game. For example, given a price of $7.00 
in period t would result in a quadratic decrease in repeat 
customers in period t. If the price in period t - 1 had been 
$8.00, the same percentage increase in price over the $8.00 base 
would result in the same loss of repeat customers. Similar 
effects occur for both advertising and R&D. This is the main 
reason why the optimal sequences of decisions in the dynamic 


analysis change from quarter to quarter. 


The method of determining potential profit for each quarter 
is particularly important. Actual profits were determined for 
each quarter by substituting the real price, advertising, R&D, 
sales and inventory into the expression to measure actual 
profitability. Determining potential profitability for any 
quarter was more complicated. The sequence of four price- 
inventory decisions that maximized the contribution measure over 
the decision horizon was found relative to each quarter by taking 


the previous quarters conditions as a starting point. Then the 


146 


summed potential profitability relative to a quarter was compared 
to the equivalent potential profitability of the succeeding quarter 
and the difference attributed to the specific quarter under 
consideration. Since both sequences of decision are optimal with 
respect to the starting quarter, the difference represents the 


potential profits for that quarter. 


The researcher assumed that the competitors' decisions 
regarding price, advertising, R&D and inventory were known rather 
than use the estimates supplied by firms. In the game, each firm 
estimated for each quarter each of these decision variables for 
its competitors. The average percentage error was small; it 
varied by firm but was well below eight percent which was not 
enough to effect results. By using the actual decisions rather 
than the estimates, the data bases supporting the interactive 
search procedures were cut by four fold and estimates of potential 


profitability were more consistent. 


Certain parameter (C,) values were required to use the 
measure of effectiveness for both the Foreign and Domestic Market. 


Cj (Advertising multiplier)= 1 for both markets. 


This has the effect of charging advertising expenditure 
against the period in which they are incurred. In the game the 
effects of advertising and R&D are smoothed in such a way that 
there is substantial effects of a present expenditure on future 
periods. However, in the reports that firms received both 
advertising and R&D expenditures were recorded as expenses for 
the period in which the decision was made. Therefore, this 
procedure was adopted even though the results of advertising and 
R&D expenditures are, in a sense, a capital investment. 

Co (R&D multiplier)= 1 for Domestic Market 

0 for Foreign Market 
This has the effect of charging all R&D expenditures against 


the domestic branch of the company as was done in the game. 
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Because R&D expenditures determine the product quality in both 
markets, this introduces very slight inaccuracies in each analysis. 
These inaccuracies are not significant enough to justify the 
substantial increase in search time and money which would be 
required to eliminate this slight inaccuracy. 

C3 (Inventory carrying cost)= $0.10 per unit with 
a doubling of this cost in excess of 375,000 in the Domestic Market 
and 35,000 in the Foreign Market. These were the actual values 
and breakpoints used in the game; therefore, they are appropriate 


parameters to use in the contribution measure. 


Reliability 

Procedural reliability relates to the ability to duplicate 
results in repeated application of the procedure. The procedures 
for determining profit potentials are more objective than 
procedures used in some experiments. Meister and Rabideau ( ) 
have noted: 


",..objectivity and subjectivity represent 
a continuum, not a diochotomy...One 
objection to using subjective judgments 
as measures is that they tend to be 
unreliable...Lack of reliability threatens 
internal validity." 

The procedures used here are more reliable because the 
measure of effectiveness was quantifiable and, therefore, did 
not require subject judgment to evaluate whether one series of 
decision was better than another. It was possible to determine 


profit potentials with a high degree of confidence. 


Finding the 96 profit potentials required 14 months. Ten 
months were used to find the 96 sequences while 4 months were 
used for verification and accuracy checking. The interactive 
search procedure was directed from the console by the experimenter. 
Over 18,000 sequences of decisions were tested before this phase 


of the experimentation was completed. 
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Search was not ended until the experimentér was confident that 
only insignificant increase in potential profits might be found 
for any of the 96 solutions. The judgment was made using a number 
of criteria developed in living with the problem for over a year. 
During that time, the experimenter became intimately familiar 
with translating different combinations of conditions into decisions 
which would realize potential profits. As a check in 12 trials 
beyond a point where continued search did not seem feasible, 


additional work increased profits less than 1/50 of 1 percent. 


On the whole, the search procedures produced sufficiently 
precise standards of comparison. Because the technique did not 
require any appreciable simplification; potential profits are 
directly comparable to actual profits. The procedures were 
applied consistently in both markets so that no bias was 
introduced. Since exhaustive search was prohibitive, results 
can not be proven optimal; this accuracy, however, is more than 


adequate for purposes of this study. 
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5.4 Chapter Summary 


This chapter documents the measures of performance and methods 
of analyses. Implication for procedural validity are examined to determie 
the quality of results. The objectives of performance measurement 
and the actual measures used in this study are described. The procedure 
for measuring decision quality and the quality indices for aided and 
non-aided decision making are derived. 

As the chapter indicates the problems involved in measuring the 
quality of non-structured decision making are formidable. To the 
experimenter's knowledge, it has never been attempted for decision making 
situations of this complexity. This chapter outlines the rationale and 
procedures and problems that encountered in measuring the quality of 
executive decision making. 

The quality of the measures and procedures determines the procedural 
validity are discussed. Questionnaire data demonstrates the relevance 
and reliability of the measures. The applicability and reasonableness 
contribute to procedural reliability of the study. 

It is fair to conclude that the procedures used here produced measures 
of decision quality - the quality indices - which are accurate and reliable. 
The verifability of the procedure contributes to the objectivity of the 
measures. This is in sharp contrast to the subjective measure used in 


prior studies of simple decision making. 
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CHAPTER 6 


RESULTS OF THE EXPERIMENT 


Each of the preceeding chapters has laid the groundwork for this chapter, 
The credibility of the results of an experimental study is based on the quality, 
care and thoroughness with which the entire study was conceived, designed and 
conducted. Prior chapters have established the quality of the study; unusual 
features contribute to the exceptionally high external, internal and procedural 
validity. Consequently, the results of the study reported in this chapter can 
be viewed with a high degree of confidence. 

This chapter presents the results of the study. The organization and con- 
tents of the chapter are shown in Figure 6.1. 
6.1 Review 


6.1.1 Hypotheses 
6.1.2 Statistical Tests 


6.2 Test of Effectiveness Hypotheses 


6.2,1 Decision Quality 
6.2.2 Decision Consistency 
6.2.3 Decision Improvement 


6.3 Test of Efficiency Hypotheses 


6.3.1 Resource Use 
6.3.2 Resolution Time 


6.4 Decision Aids 


6.4.1 Statistics of Use 
6.4.2 Quality of Aids 


6.5 Conclusions 


Figure 6.1 Contents of Chapter 6 


The first section reviews hypotheses and discusses the reasons for using 
nonparametric statistical tests. Later sections present the data and hypotheses 
tests. Information concerning the use of decision aids is related to each 
of the hypotheses in this study. The final section draws conclusions for the 


entire study. 
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6.1 Review 


This section reviews the hypotheses and the statistical 
tests that are used in the study. Two types of nonparametric 
tests are used. Their appropriateness is determined by the 
design of the experiment and the level of measurement achieved. 
The hypotheses are reviewed first because they indicate the 
comparative nature of the study and the need for tests for 


related samples. 


6.1.1 Hypotheses 


The hypotheses presented in Chapter 2 are reviews prior to 


the formal tests presented in this chapter. 


Effectiveness Hypothesis 


The effectiveness hypotheses are: 


For non-structured managerial tasks, DSS-aided 
decision making will result in decisions whose: 

(H1) - quality is higher 

(H2) - quality is more consistent 

(H3) - quality has a higher rate of improvement 
than non-aided decision making in similar, 


controlled circumstances. 


Each of these hypotheses relates to the level or pattern of 
decision quality represented by the table of quality indices 
found in Chapter 5. 


Efficiency Hypotheses 


The efficiency hypotheses are: 


For non-structured managerial tasks, DSS-aided 
decision making will result in: 

(H4) - lower resource use 

(H5) - shorter resolution time 
than non-aided decision making in similar, 


controlled circumstances. 
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These hypotheses relate to different aspects of decision 
time - contact time and elapsed time. Resource use is a measure 
of the man-hours used in DSS-aided and non-aided decision making 
and, therefore, represents contact time. Resolution time is a 
measure of the elapsed time from the beginning of the decision 


period to the final choice of an alternative. 
6.1.2 Statistical Tests 


The use of statistical tests is related to the level of 
measurement achieved and the design of the experiment. An interval 
level of measurement for quality indices was obtained by the 
analytical procedures. Interval data allows the researcher to 
use either parametric or nonparametric statistical tests. An 
ordinal level of measurment was achieved for man-hours and 


resolution time. This requires the use of nonparametric tests. 


Certain assumptions must be met to use parametric tests. 
The requirements for the use of parametric tests are: 
(1) Observation be independent, 


(2) Observation be drawn from normally 
distributed population, and 


(3) Population must have the same variances. 
The meaningfulness of the result of a parametric test depends 
hevily on the validity of these assumptions. These requirements 
are highly restrictive and cannot be assumed with any confidence 
in this study. Therefore, nonparametric tests are used for 
testing hypothesis. Siegel (102) claims 


"A nonparametric statistical test is a test 
whose model does not specify conditions 
about the parameters of the population 
from which the sample was drawn. Certain 
assumptions are associated with most 
nonparametric statistical tests, but these 
assumptions are fewer and much weaker than 
those associated with parametric tests." 
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Nonparametric tests are useful when the assumption and 
requirements of parametric tests are unrealistic for the 
data, which is the case in this experiment. Because non- 
parametric tests require fewer assumptions, external validity 


and generality of results are increased. 


The nonparametrical statistical tests used in this study 
are the appropriate testgfor two related samples involving 
means, variances and differences for aided and non-aided decision 
making. The use of these tests is noted in Sellitz (98): 

"The two-sample statistical tests are 
used when the researcher wishes to 
establish whether two treatments are 
different, or whether one treatment 

is better than another...In each case, 
the group which has undergone the 
treatment is compared with one which 
has not, or which has undergone a 
different treatment." 

Comparative tests for two related samples are appropriate 
for data involving subjects that act as their own controls or 
for matched-pairs experimentation because these tests do not 


assume that all pairs are drawn from the same population. 
The nonparametric tests used in this study are the: 


(1) Wilcoxon Matched-Pairs Signed Rank Test 
(2) The Sign Test 


Complete documentation is given in Siegel (102). 
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6.2 Tests for Effectiveness Hypotheses 


Each hypothesis will be tested using the Wilcoxon Matched- 
Pairs Signed-Rank Test. This nonparametrical test for related 
samples is the appropriate test to use because the quality 
indices can be ranked on an ordinal scale both within and between 
teams; and the parent population is assumed to be non-normal 
which precludes parametrical tests. The procedure for using 
the Wilcoxon test is given in Siegel (102). This method is 
fairly standardized and adapts nicely to the work that follows. 


The appropriate data for these hypotheses are the quality 
indices shown in Table ‘6{1L The table shows the quality 
indices for each firm for both the non-aided and aided decision 
making conditions. The table indicates that there were six 
matched-pairs involved in the experiment - Firms 1 to 6. Each 
group was involved in eight replications of the experiment; 
that is each quarter amounted to a replication of the experiment 
under different sets of conditions. Therefore, each quality 


index represents a "raw" data point. 


Summary statistics are shown in Table 642. These statistics 
are based on the quality indices for quarters 13 to 19 because 
the Marketing Decision System was not available for use in 
quarter 12. Each set of statistics are used in the three 


tests for effectiveness hypothesis. 
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TABLE 6,1 
Quality Indices 


Quality Indices for Non-Aided Decision Making 


Domestic Market -— Control Condition 


Quarter Firm l Firm 2 Firm 3 Firm 4 Firm 5 Firm 6 
12 0.985 0.974 0.989 0.981 0.988 0.970 
13 0.989 0.961 0.997 0.992 0.976 0.988 
14 0.925 0.890 0.918 0.919 0.936 0.956 
15 0.580 0.617 0.603 0.623 0.613 0.521 
16 0.855 0.807 0.971 0.817 0.820 0.870 
17 0.937 0.932 0.947 0.950 0.912 0.940 
18 0.981 0.939 0.853 0.943 0.958 0.872 
19 0.935 1.000 0.732 0.680 1.000 0.792 


Quality Indices for Aided Decision Making 


Foreign Market - Experimental Condition 


Quarter Firm l Firm 2 Firm 3 Firm 4 Firm 5 Firm 6 
12 0.748 0.742 0.706 0.726 0.759 0.769 
13 0.791 0.819 0.795 0.809 0.798 0.761 
14 0.937 0.954 0.970 0.945 0.943 0.975 
15 0.876 0.920 0.995 0.981 0.971 0.643 
16 0.988 0.850 1.000 1.000 0.959 0.978 
17 0.994 0.992 1.000 1.000 0.981 1.000 
18 0.993 0.961 1.000 1.000 0.986 1.000 
19 0.978 0.990 1.000 1.000 0.910 1.000 


i 
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TABLE 6,2 


Summary Statistics for Quality Indices 


Statistics for Non-Aided Decision Making 


Domestic Market - Control Condition 


Firm Mean | Median Variance Std. Dev. 
1 0.886 0.935 0.0201 0.142 
2 0.878 0.932 0.0170 0.130 
3 0.860 0.918 0.0208 0.144 
4 0.847 0.919 0.0208 0.144 
5 0.888 0.936 0.0181 0.134 
6 0.846 0.872 0.0251 0.159 


Statistics for Aided Decision Making 
Foreign Market - Experimental Condition 


Firm Mean Median Variance Std. Dev. 
1 0.936 0.937 0.00595 0.0772 
2 0.927 0.954 0.00462 0.0680 
3 0.966 1.000 0.00579 0.0761 
4 0.962 1.000 0.00497 0.0705 
5 0.936 0.959 0.00434 0.0659 
6 0.908 0.978 0.00211 0.1452 
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6.2.1 Decision Quality Hypothesis 


(Hl) - For non-structured managerial tasks, 
DSS-aided decision making will result 
in decisions whose quality is higher 
than non-aided decision making in 


similar, controlled circumstances. 


This hypothesis can be tested by establishing the null 
hypothesis: 
H,: The decision quality for the experimental 
condition does not differ from the 
decision quality for the control condition. 
and the alternative hypothesis: 
H,° The decision quality for the experimental 
condition is better than the decision 


quality for the control condition. 


The appropriate data for this test are either the means or the 
medians of the quality indices for each firm for both the control 
and experimental conditions. The use of means to measure average 
quality for each team is more appropriate and, in fact, is a better 
measure of decision quality because it adequately captures the 


range of quality for a variety of circumstances that existed during 


the experiment. 


The data is shown in Table 6,3. The means for each firm are 
identical to those shown in Table 6.2 These means represent 
the average quality for each firm for DSS~aided and non-aided 


decision making from quarter 13 to 19. 
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TABLE 6.0 


Index of Mean Quality 
Quarters 13 - 19 


For Each Firm 


Index of Mean Quality Index of Mean Quality 
Firm for DSS-Aided Decision for Non-Aided Decision 

Making Making 

Firm l 0.936 0.886 

Firm 2 0.927 0.878 

Firm 3 0.966 0.860 

Firm 4 0.962 0.847 

Firm 5 0.936 0.888 

Firm 6 0.908 0.846 

Result: 


The index of mean quality for DSS-aided decision making 
is higher for each firm than the comparative index for non-aided 
decision making. The Wilcoxon matched-pairs, signed-rank test 
accepts the alternative hypothesis at least at the 0.025 level. 
This result implies that quality for DSS-aided decision making 
is significantly better than the quality for non-aided decision 


making. 
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6.2.2 Test of Decision Consistency 


(H2) - For nonsstructured managerial tasks, 
DSS-aided decision making will result 
in decisions whose quality is more 


consistent than non-aided decision 


making in similar, controlled 


circumstances. 
This hypothesis can be tested by establishing a null hypothesis: 


H,: The consistency of quality for the 
experimental condition does not differ 
from the consistency of quality for 
the control condition. 

and the alternative hypothesis is: 

H,: The consistency of quality is better 
(shows smaller variations) for the 
experimental condition than the 
consistency of quality for the control 


condition. 


To test this hypothesis requires a measure comparing variations 
in quality. An adequate indicator of the spread of the quality is 
the variance or standard deviation of each quality indices. 

Table 6,2lists the measures for both DSS-aided and non-aided 
decision making for each firm. The data chosen for this test are 

the standard deviations shown in Table 6,4. The standard deviations 
represent this consistency of quality for each firm for DSS-aided 


and non-aided decision making from quarters 13 to 19. 
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TABLE 6.4 


Standard Deviation of Quality 
Quarters 13 - 19 


For Each Firm 


————————— oO ———————————————————— ee 


Standard Deviation Standard Deviation 
Firm for DSS-Aided Decision for Non-Aided Decision 

Making Quality Making Quality 

Firm l 0.0772 0.142 

Firm 2 0.0680 0.130 

Firm 3 0.0761 0.144 

Firm 4 _ 0.0705 0.144 

Firm 5 0.0659 0.134 

Firm 6 0.1452 0.159 

Se Sa a 

Result: 


The table shows that the standard deviation for each team 
is smaller in the experimental condition than in the control 
condition. The Wilcoxon matched-pairs, signed-rank test accept 
the hypothesis at a level of significane of at least 6.025. 
This implies that DSS-aided quality is significantly more 


consistent than non-aided decision quality. 
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6.2.2 Test of Rate of Decision Quality Improvement 


(H3) - For non-structured managerial tasks, 
DSS-aided decision making will result 
in decisions whose quality has a higher 
rate of improvement than non-aided 


decision making in similar, controlled 
circumstances. 


This hypothesis can be tested by establishing a null hypothesis: 

H.: The rate of improvement for the 
experimental condition is equal to 
(or less than - i.e.negative) the 
rate of improvement for the control 
condition. 

and the alternative hypothesis: 

H,: The rate of improvement for the 

experimental condition is greater than 


the rate of improvement for the control 


condition. 


The basic problem in testing this hypothesis stems from the 
fact that all teams experienced large fluctuations in decision 
quality. These variations are both increases and decreases in 
quality over the previous quarters and are much more pronounced 
in the Domestic Market than in the Foreign Market. This problem 
can be minimized by using the net rate of change over the game 


as a measure of change in quality levels for each firm. 


To calculate the net rate of change, the first step is to 
determine quarterly differences in decision quality for each team 
for every quarter. These differences are given in Table é.& 

A plus indicates an increase in quality over the previous quarter 
while a minus indicates a reduction in quality over the previous 
quarter. The magnitude of the signed numbers indicate the size 
of the increase or reduction in quality relative to the previous 


quarter. 
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Quarter 
From-To Firm 1 Firm 2 Firm 3 Firm 4 = Firm 5 
12 - 13 +0.004 -0.013 +0.008 +0.011 -0.012 
13 - 14 -0.064 -0.071 -0.079 -0.073 -0.040 
14 - 15 -0.345 -0.273 -0.315 +0.296 -0.323 
15 - 16 +0.275 +0.190 +0.368 +0.194 +0.307 
16 - 17 +0.082 +0.125 -0.024 -0.133 -0.008 
17 - 18 +0.044 +0.007 -0.094 -0.007 +0.046 
18 - 19 -0.046 +0.061 -0.121 -0263 +0.042 
Rate of Change in Decision Quality 
Foreign Market - Experimental Condition 
hg i 
Quarter 
From-To Firm 1 Firm 2 Firm 3 Firm 4 #£Firm5 
12 - 13 +0.053 +0.047 -0.011 -0.083 +0.039 
13 - 14 +0.143 +0.135 +0.175 +0.136 +0.145 
14 - 15 ~0.058 -0.034 +40.025 +0.036 +0.028 
15 - 16 +0.112 -0.070 +0.005 +0.019 -0.012 
16 - 17 +0.046 +0.142 .000 0.000 -0.022 
17 - 18 -0.001 -0.031 .000 0.000 +0.005 
18 - 19 -0.015 -0.029 .000 0.000 -0.076 


TABLE 6.5 


Changes in Decision Quality 


Rate of Change in Decision Quality 


Domestic Market - Control Condition 
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Firm 6 
+0.018 
-0.032 
-0.435 
+0.349 
+0.070 
-0.068 
-0.080 


The signed numbers in Table 6-0 are true rates of change for 
decision quality over each quarter because they are, in fact, the 
slope of the decision quality graphs. The net rate of change can 
then be calculated for each team by determining the algebraic sum 
of the increases in quality, less all decreases in quality. If this 
sum is positive, the decision quality would show an increase, but if 


the sum is negative, the decision quality would show a decrease. 


To compare the rates of change in decision quality between 
the two markets, requires computing the algebraic sum for the 
net difference score for the Wilcoxon test. This comparison is 


given in Table 6,6: 
TABLE 6,6 


Net Rates of Change of Decision Quality 


Net Rate of Change Net Rate of Change Signed 


Firm for DSS-Aided for Non-Aided Difference 
Decision Making Decision Making 

Firm l +0.227 -0.054 +0.281 
Firm 2 +0.113 +0 .039 +0.074 
Firm 3 +0.205 -0.245 +0.450 
Firm 4 +0.191 -0,327 +0.518 
Firm 5 +0 .068 +0 .024 +0 .044 
Firm 6 +0.239 -0.196 +0.435 
Result: 


All of the signed differences are positive. This could only 
occur if the rate of improvement of decision quality is greater for 
the experimental condition than for the control condition. The 
Wilcoxon matched-pairs signed-rank tests reflects this fact by 
accepting the hypothesis at the 0.025 level. The analysis and 
hypothesis test suggest that the rate of improvement in decision 
quality is significantly higher for DSS-aided decision making than 


for non-aided decision making. 
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6.3 Tests for Efficiency Hypotheses 


Each hypothesis will be tested using the Sign Test, This 
nonparametrical test for related samples is the appropriate test 
to use because the data can be ranked on an ordinal scale both 
within and between teams while the parent population is assumed 
to be non-normal. thus precluding parametrical tests. The procedure 
for using the Sign Test is given in Siegel (102). This test adapts 
nicely to the work that follows. 

The appropriate data for these hypotheses are required for man- 
hours and resultion time making foreign and domestic marketing deci- 
sions. Tables 6.7 and 6.8 show the data for each firm for both the 
non-aided and aided decision making conditions. The tables are set 
up to show that there were six matched-pairs involved in the experi- 
ment. Each group was involved in eight replications of the experiment; 
that is, each quarter amounted to a replication of the experiment 
under different sets of conditions. Therefore, each sign represents 
a "raw" data point. 

Summary statistics are shown in Tables 6.7 and 6.8. These statis- 
tics are based on data for quarters 13 to 19 because the Marketing 


Decision System was not available for use in quarter 12. 
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6.3.1 Resource Use Hypothesis 


(H4) - For non-structured managerial tasks, 
DSS-aided decision making will result 
in lower resource use than non-aided 


decision making in similar, controlled 
circumstances. 


This hypothesis can be tested by establishing a null hypothesis: 

Hn? The number of man-hours required for 
Foreign Marketing decisions is equal to 
the number of man-hours required for 
Domestic Marketing decisions 

and the alternative hypothesis: 

Ha: The number of man-hours required for 
Foreign Marketing decision is less than 
the number of man-hours required for 


Domestic Marketing decision. 


The data for this test are shown in Table 6.% These entries 
are estimates of the man-hours used in making the Domestic and 
Foreign Marketing decisions. Since an ordinal level of measurement 
is achieved, the appropriate statistical test is the sign test for 
related samples. These data can, at best, be only partially 
ordered. The most important requirement for use of sign test is 
that it must be possible to rank the two values of each pair with 
respect to each other. This requirement was met for the data. 
Thus, the information contained in the estimates is saved by 


expressing the difference as a sign. 
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TABLE 6,'7 


Estimated Man-Hours for Marketing Decisions 


Quarter Firm 1 Firm 2 Firm 3 
Dom. For. Sign Dom. For. Sign. Dom. For. Sign 

i ER 

12 3.7 4.0 - 4.8 3.8 + 3.0 + 

13 228 Anz + 0.5 0.5 = 1.6 2.3 - 

14 2.0 1.3 + O58 05 = 13 14 - 

15 3.5 1.7 + Oe | 1.5 + 2. “240 + 

16 is? is2 + 2.0 3.8 - Le; 1.3 = 

17 Ie? -155 + 1.8 1.6 + a, 0.7 + 

18 120°: JigZ - 1.9 i373 + i 0.7 + 

19 1.7. 1.0 + 0.6 lige - Ne 0.8 + 
a 
Quarter Firm 4 Firm 5 Firm 6 

Dom. For. Sign Dom. For. Sign Dom. For. Sign 

Se ag as i lr ee 

12 304: Bak + 4.0 3.0 + Zea Lad + 

13 2.4 1.9 + 3.0 2.3 + 1.2 1.6 - 

14 128.- eZ + 1.7 2.0 - ke? 13 + 

15 2.7. 1.3 + 330-257 + 2.8 2.4 + 

16 1.9 1.0 + 240". Ae2 + 1 1.4 + 

17 1.6 2.0 = 1.6 1:2 + 1.6 1.0 + 

18 1.1 1.4 - Led "Led + 1 ded = 

19 1.0 0.7 + 1.3 0.9 + 1.0 0.8 + 
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The signs to the right of each matched-pair indicate: 


Sign Meaning 
+ Man-hours required for Domestic Marketing 


Decisions were greater than those required 
for Foreign Marketing Decisions 


= Man-hours required in both markets were 
equal 


- Man-hours required for Domestic Marketing 
Decisions were less than those required 
for Foreign Marketing Decisions. 


Result: 


The data shows a preponderance of plus signs which indicate 
that in most quarters the number of man-hours devoted to Domestic 
marketing decision making was greater than the number of man-hours 
devoted to Foreign marketing decision making. The formal sign 
test confirms the fact by accepting the hypothesis at the 0.05 
level of significane. These results suggest that teams required 
fewer man-hours to make marketing decisions in the Foreign Market 


than in the Domestic Market. 
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6.3.2 Resolution Time 


(H5) - For non-structured managerial tasks, 
DSS-aided decision making will result 


in shorter resolution time than non-aided 


decision making in similar, controlled 
circumstances. = 


This hypothesis can be tested by establishing a null hypothesis: 

H,: The clock time to final choice for 
Foreign Marketing decisions is equal 
to the time required for Domestic 
Marketing decisions 

and the alternative hypothesis: 

H,: The clock time to final choice for 
Foreign Marketing decision is less than 
the time required for Domestic Marketing 


decision 


The data for this test are shown in Table 6.&These entries 
are estimates of resolution time for the Domestic and Foreign 
Marketing decisions. The appropriate statistical test is, once 
again, the sign test for related samples because this data can 


be only partially ordered. 


The signs to the right of each matched-pair indicate: 


Sign Meanin 
ign Heaning 
+ Resolution time for Domestic Marketing 


decision was greater than Foreign 
Marketing decisions 


= Resolution time in both markets were 
equal 


- Resolution time for Domestic Marketing 
decisions was less than Foreign 
Marketing decisions. 
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TABLE 6.8 


Resolution Time for Marketing Decisions 


Firm 4 Firm 5 Firm 6 
For. Sign. Dom. For. Sign. Dom. For. Sign. 


Dom. 


Quarter 
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Result: 


The data shows a greater number of plus signs. This indicates 
that Domestic Marketing decisions require more time to resolve. 
The formal sign test confirms this fact by accepting the hypothesis 
at the 0.05 level of significance. These results suggest that 
teams arrive at Foreign marketing decisions faster than in the 


Domestic Market. 
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6.4 Decision Aids Use 


The Interactive Systems available to each firm were: 
MDS - Marketing Decision System 
FPS - Financial Planning System 
SAS - Statistical Analysis System 
MDS could only be used to aid the decision maker in the 
Foreign Market; its use was prohibited in the Domestic Market. 
FPS could be used to aid making financial decisions in both markets. 


SAS could be used to develop models and analyses in either market. 


In the true technical sense marketing decision in the 
Domestic Market was not strictly unaided. Even though FPS and 
SAS were available, the Domestic Market is considered non-aided 
in that no true Decision Support System was provided to directly 
aid non-structured decision making. Specifically, no formal 
system was available in the Domestic Market in which a decision 
maker could work directly with the decision variables under his 
control. In other words, formal models were not provided in 


SAS; only the capability to perform statistical analyses. 


In contrast, MDS was a system in which a formal interactive 
simulation model was available to evaluate alternative strategies 
concerning the firm's decision variables and the effects of 
competitive and economic changes. It dealt directly with the 
firms marketing decision variables; Price, Advertising, R&D and 


Inventory for the Foreign Market. 


6.4.1 Statistics of Use 


The use of each package was measured in the following way. 
For MDS the number of completed runs were tallied for the 
Modeler (M) option and the Future (F) option. For FPS the 
number of completed runs were tallied for the Planner (P) option 
and the Forward (F) option. The Forward option results are not 
reported because this feature was used only once. For SAS the 
number of transformations (T) entered, plots (P) requested and 
regression (R) runs completed were tallied by team by quarter. 
Only the regression results are reported in the Table : 
A dash indicates that the package was not used during the quarter. 
All firms attempted runs which were not completed due to either 
computer system failures and/or mistakes made by the firms. In 


either case runs were aborted; these attempts are not tallied. 


MDS was not available for use during quarter 12 due to 
technical difficulties; therefore, the dash indicates no use - 
not because firms neglected to use the package but simply because 
it was not available. MDS was available from Quarter 13 through 
Quarter 19. Thus, some of the early runs, especially those in 
Quarter 13 may have been used by the firms to familiarize them- 
selves with MDS operations. Figures 6,2to 6,6 compare the 
fequency use of these systems. The frequency of use is a rough 
indicator of the decision makers' need for the particular support 
package. Given this background it can be seen that firms used 
both MDS and FPS extensively. In fact, one member of each four- 
man team spent the majority of the three-hour console session at 
the terminal. No console session was less than three hours in 
length and many sessions were extended for the convenience of 


the teams. 
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Figure 6.2 Use of Marketing Decision System 
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Figure 6.3 Use of Marketing Decision System 
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Figure 6,4 Use of Marketing Decision System 
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Figure 6,6 Use of Statistical Analysis System 
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Regression Option 
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6.4.2 Quality of Aid 


Figures 6.20 Gshowed that the MODELER and FUTURE option of 
MDS were used extensively. The only question which remains to 
be answered is whether this use directly aided the Foreign 
marketing decision making. Some or all of the runs may have been 
"playing with the system", although there is a low probability 
that this occurred. Each firm indicated on the questionnaire 
that the MDS was used as an aid in making marketing decisions in 
the Foreign Market. This response, however, does not detail the 


quality of the aid provided by the MDS. 


A better indication can be obtained by comparing the actual 
Foreign marketing decisions of Price, Advertising, R&D, and 
Inventory with those that were used as input to MODELER. Because 
decisions were not finalized until the end of the console sessions, 
a direct comparison indicates the system was used to aid in making 


the decisions for that quarter. 


Table 6,9provides data on the comparisons of actual marketing 
decisions for the quarter and the closest complete MDS runs that 


were found in the trace. 


TABLE 6.9 


Degree of Correspondence between MDS Runs and Actual Decisions. 


ener eeepc nen ee A RS GEN A A gree oeenernyeemeneene 
ae epee megane cient gn aR gE EL APR LL eee 


Firm Three Variables Two Variables One Variable Qtr 
P/A/R P/A P/R A/R P AR Used 
a 
Firm 1 4 1 1 1 - - - 7 
Firm 2 4 2 1 - - - - 7 
Firm 3 2 1 1 - 2 - il 7 
Firm 4 3 - 2 1 - - I 7 
Firm 5 3 1 - 1 2 -- 7 
Firm 6 Z Z - ~ - - 5 
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It should be clear from this exhibit that all decisions were 
DSS-aided. The degree of aid which the system provided is shown 
by the number of exact and partial correspondences between MDS 


runs and actual decisions of Price Advertising and R&D. 
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6.5 Conclusions 


As stated earlier, for decision making to be judged more 
effective required the acceptance of each of the decision quality 
hypotheses. Each hypothesis has been formally accepted in this 
section. In particular, decision making in the Foreign Market 
resulted in decisions whose: 

(H1) quality is higher 
(H2) quality is more consistent 


(H3) quality has a higher rate of improvement 


Therefore, it is fair to conclude that decision making in 
the Foreign Market is significantly more effective than in the 
Domestic Market. In addition, the hypothesis (H4, H5) related to 
decision efficiency were also accepted. Significantly fewer 
man-hours were devoted to making Foreign marketing decisions 
than to Domestic marketing decisions. Foreign marketing decisions 


also require less resolution time. 


This compound result is highly significant. The efficiency 
and effectiveness of decision making in the Foreign Market are 


substantially better. 


Alternative Explanations of Derived Results 


There are several factors which had the potential to 
influence the results and thus serve as the basis for alternative 
explanations of the hypotheses. These factors are discussed in 


this section. 


Concerning hypotheses 1, 2, 3, there are two possibilities 
to consider. One factor is that there might have been less 
difficulty facing the decision makers in the Foreign Market than 
in the Domestic Market. Considerable detail related to the game 
is needed to evaluate this contention. A complete discussion 


is provided in Appendix 
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The conclusions reached in the appendix can be summarized 
succinctly. In no quarter was the difficulty in the Foreign Market 
less than the difficulty faced in the Domestic Market. On the 
contrary, the Foreign Market was considerably more difficult to 
play. It is fair to say that the difficulty in the Foreign Market 
completely dominated the difficulty faced in the Domestic Market. 
In other words, the lack of comparative difficulty can be ruled 
out as an explanation of increased decision effectiveness in the 
Foreign Market. In fact, the higher difficulty in the Foreign 
Market provided a more rigorous test for the three hypotheses 


by biasing the results toward rejecting the hypotheses. 


Another factor to consider is that there might have been 
transfer of knowledge between the markets. This was a serious 
concern during the design of the experiment. Therefore, a 
number (7) of different questions were asked in the questionnaire 
to determine if transfer of knowledge took place. Results shown 
in Chapter 3 can be summarized by stating that 

(1) No decision maker on any team suspected, even 

remotely, that the markets were structurally 
identical. 

(2) Every decision maker felt that any transfers which 


may have occurred were not very significant. 


There is a paradox here. Any transfer which may have occurred 
would, more than likely, have been from the Foreign to the Domestic 
Market rather than the reverse because results were superior in 
the Foreign Market. Thus, transfers of knowledge would have biased 


the experiment, once again, toward rejecting the hypothesis. 


Concerning hypothesis H4, one factor to consider is that the 
relative size of the markets and potential profits might have 
influenced the number of man-hours devoted to decision making in 
each market. The Domestic Market was larger than the Foreign 
Market, therefore, more man-hours might be allocated to the Domestic 


marketing decisions at the expense of the Foreign marketing decisions. 
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There is no doubt that all firms were cognizant of the 
difference in market sizes and that this had an influence on the 
man-hours allocated to each. This fact is apparent from responses 
to both the questionnaire and the interview why fewer man-hours were 
devoted to making marketing decisions in the Foreign Market relative 
to the Domestic Market, each firm stated, categorically, that the 
main reason for the reduction in time was the availability of the 
Marketing Decision System. In addition, they also expressed the 
opinion that the number of hours would have been approximately 
equal had no Decision Support System been available. The reason 
given for this was that the Foreign Market was more attractive 
because: 

(1) the profit margins in the Foreign Market were higher and 

(2) the Foreign Market was growing faster than the Domestic 


Market, 


Although it is not possible to demonstrate that the only 
reason fewer man-hours were devoted to Foreign marketing decisions 
was the availability of the Decision Support System, there are 


sufficient reasons to believe that this was the primary factor. 


To summarize, the preceding discussion has shown that 
factors which had the potential to influence the outcome of the 
experiment had either a negligible effect or actually strengthened 
the results by making the acceptance of the hypotheses more 
difficult. 
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APPENDIX A 
DATA INPUT FORM 


MARKETING FORECASTS AND PLANS 


Domestic Mkt. ; 
Se eS OTR, OTR» QTR. 
Industry 's a ys Tae oi 
Total Mkt. 


Firm's 
Pot. Mkt. 


Firm's 
Sales Plan 


Foreign Mkt. 


Industry's 
Total Mkt. 


Firm's 
Pot. Mkt. 


Firm's 
Sales Plan 
PRODUCTION/CAPACITY PLANS 

Total Prod. 

Dom. Prod. 

For Ship. 

Machine Cap. 

Labor Force 


eT ETRE SERENE 


INFORMATION ESTIMATES 


QTR. 


DECISIONS THIS QTR. 


Marketing Decisions 
1. Domestic 
Price 
2. Foreign 
Price 


Domestic 
Promotion 


Foreign 
Promotion 
5. R&D 
Expenditure 


—____,— Production Decisions 


6. Units 


Produced 
7. Units 
Shipped 
8. Machine 
Capacity 
9. Labor 
Force 


ame 
ry 


Finance Decisions 


10. Cash 
Domestic Mkt. GNPD SEAS Remitted 
Price Promotion Production Inventory 
Firm 1 . 11. Domestic 
Firm 2 Securities 
Firm 3 
Firm 4 12. Foreign 
Firm 5 Securities 
Firm 6 . 
Foreign Mkt. GNPF SEAS 13. Domestic 
Loans 
Price Promotion R & D Inventory Sa 
Firm 1 14. Foreign 
Firm22 Loans 
Firm 3 
Firm 4 15. Shares 
Firm 5 Outstanding 
Firm 6 
16. Dividends 


APPENDIX B 


MARKET SUMMARY REPORT 


Quarter il 


DOMESTIC MARKET REPORT 


MARKET 1 GNP=503.30 TOTAL UNITS = 3113000. 


LOANS ACCRUED TAXES 


ooo0o0oo 


496200 
496200 
496200 
496200 
496200 
496200 


250000 
200000 
210000 
170000 
130000 
130000 


FIRM UNITS PRICE LOST SALES “THIS %TOTAL 
1 532000. 6.00 94000. 17.1 14.9 
2 508000. 6.00 79000. 16.3 14.2 
3 464000. 6.00 96000. 14.9 13.0 
4 609000. 6.00 90000. 19.6 17.0 
5 507000. 6.00 105000. 16.3 14.2 
6 493000. 6.00 96000. 15.8 13.8 
FOREIGN MARKET REPORT 
MARKET 2 GNP=181.50 TOTAL UNITS = 464000. 
GNP PROJECTIONS: 191.0 196.0 202.0 209.0 
FIRM UNITS PRICE LOST SALES “THIS “%ZTOTAL 
1 65000. | 8.00 44000. 14.0 1.8 
2 83000. 8.00 58000. 17.9 Zea 
3 91000. 8.00 105000. 19.6 245 
4 80000. 8.00 67000. L752 202 
5 71000. 8.00 88000. 15.3 2.0 
6 74000. 8.00 44000. 15.9 2.1 
SUMMARY INFORMATION ON INDUSTRY 
FIRM CASH SECURITIES INVENTORY OTHER 
1 1541294 200000 2354999 10350000 0 
Z 1548859 200000 2354999 10375000 0 
3 1535855 200000 2354999 10325000 0 
4 1549932 200000 2354999 10350000 0 
5 1540500 200000 2354999 10350000 0 
6 1532608 200000 2354999 10350000 0 
FIRM COMMON STOCK NET WORTH DIVIDENDS DOMESTIC ADV. FOREIGN ADV. R&D 
1 10000000 3950093 50000 90000 
2 10000000 3952658 50000 80000 
3 10000000 3969654 50000 90000 
4 10000000 3958731 50000 100000 
5 10000000 3949299 50000 130000 
6. 10000000 3941407 50000 100000 
FIRM STOCK PRICE MARKET VALUE TOTAL NET 
1 27.17 13585342 13950093 
2 28.85 14426055 13952658 
3 31.26 15631568 13969654 
4 28.94 14468292 13958731 
5 29.26 14630744 13949299 
6 28.31 14154090 13941407 


APPENDIX C 


REPORT FOR QUARTER 11 


DOMESTIC PROFIT AND LOSS 


SALES REVENUE 
COST OF GOODS SOLD 
GROSS MARGIN 


MANUFACTURING OVERHEAD 

OVERTIME 

HIRING 

OVERHEAD EXPENSE 

SHIPPING 

PROMOTION 

R&D EXPENSE 

SELLING AND ADMINISTRATIVE EXPENSE 
OPERATING EXPENSES 

TOTAL EXPENSES 


OPERATING INCOME 
INTEREST INCOME 
NET INTEREST 


NET INCOME BEFORE TAX 
TAXES 


NET INCOME AFTER TAX 


REPORT FOR QUARTER 11 


DOMESTIC BALANCE SHEET 


CASH 

SECURITIES 

WORK IN PROCESS 

TOTAL INVENTORY 

PLANT 

DEPRECIATION 

NET BOOK VALUE OF PLANT 
INVESTMENT IN FOREIGN OPERATION 
TOTAL ASSETS 


ACCRUED TAXES 
COMMON STOCK 

NET WORTH 

TOTAL LIABILITIES 


3270000 
1907500 
57500 
39000 
80000 
61500 
20000 
100000 
200000 
159000 
479000 
1625 
1625 
395340 
1070573 
100000 
2354999 
2354999 
13000000 
2650000 
10350000 
478499 
395340 
10000000 
3958731 


1362500 


540500 


822000 


823625 


428285 


14354071 


14354071 


APPENDIX C 
REPORT FOR QUARTER 11 


DOMESTIC SOURCES AND USES OF FUNDS 


SOURCES 
CHANGES IN RETAINED EARNINGS 487550 
DEPRECIATION 325000 
CHANGES IN ACCRUED TAXES 240470 
TOTAL SOURCES 1053020 
USES 
CHANGES IN CASH 896255 
CHANGES IN INVENTORY 103500 
CHANGES IN FOREIGN INVESTMENT 53265 
TOTAL USES 1053020 
REPORT FOR QUARTER 11 
DOMESTIC RECONCILIATION OF RETAINED EARNINGS 
NET INCOME AFTER TAX 428285 
INCREASE IN FOREIGN RETENTIONS 53265 
SHIPMENTS OF GOODS 344000 
REMITTANCES 400000 
DIVIDENDS 50000 
CHANGE IN RETAINED EARNINGS 487550 
RETAINED EARNINGS BEGINNING OF PERIOD 3471181 
RETAINED EARNINGS END OF PERIOD 3958731 
REPORT FOR QUARTER 11 
DOMESTIC INVENTORY RECONCILIATION 
UNITS VALUE 

INVENTORY BEGINNING OF PERIOD 0 0 
RECEIPTS 545000 1907500 
SALES 545000 1907500 
INVENTORY END OF PERIOD 0 0 
WORK IN PROCESS 5350000 1925000 


AVATLABLE FOR SALE NEXT PERIOD 550000 1925000 


APPENDIX D 


REPORT FOR QUARTER 11 


FOREIGN I'ROFIT AND LOSS 


SALES REVENUE 
COST OF GOODS SOLD 
GROSS MARGIN 


TARIFF 

PROMOTION 

SELLING AND ADMINISTRATIVE EXPENSE 
OPERATING EXPENSES 

TOTAL EXPENSES 


OPERATING INCOME 
INTEREST INCOME 
NET INTEREST 


NET INCOME BEFORE TAX 
TAXES 


NET INCOME AFTER TAX 


16000 
10000 
62000 


2125 


REPORT FOR QUARTER 11 
FOREIGN BALANCE SHEET 


CASH 
SECURITIES 
TOTAL ASSETS 


ACCRUED TAXES 
NET WORTH 
TOTAL LIABILITIES 


640000 
344000 


88000 


2125 


100860 


479359 
100000 


100860 
478499 


296000 


88000 


208000 


210125 


109265 


579359 


579359 


APPENDIX D 


REPORT FOR QUARTER 11 


FOREIGN SOURCES AND USES OF FUNDS 


SOURCES 
CHANGES IN RETAINED EARNINGS 
TOTAL SOURCES 


USES 

CHANGES IN CASH 

CHANGES IN ACCRUED TAXES 
TOTAL USES 


53265 


43665 
9600 


REPORT FOR QUARTER 11 


53265 


53265 


FOREIGN RECONCILIATION OF RETAINED EARNINGS 


NET INCOME AFTER TAX 
SHIPMENTS OF GOODS 
REMITTANCES 


CHANGE IN RETAINED EARNINGS 


RETAINED EARNINGS BEGINNING OF PERIOD 


RETAINED EARNINGS END OF PERIOD 


REPORT OF QUARTER 11 


109265 
344000 
400000 


FOREIGN INVENTORY RECONCILIATION 


INVENTORY BEGINNING OF PERIOD 
RECEIPTS 

SALES 

INVENTORY END OF PERIOD 

WORK IN PROCESS 

AVAILABLE FOR SALE NEXT PERIOD 


UNITS 


0 
80000 
80000 

0 

100000 
10.0000 


53265 
425234 
478499 


VALUE 
0 

34 7500 

347500 
0 


435000 


435000 


Senior Executive 


Atkins, M. A. 


Blackadder, T.S. 


Brockmeier, K.H. 


Burn, J.A.S. 


Clark, D.R. 


Dickson, R.S. 


Finegan, C. 


APPENDIX E 


Profile of Participants 


Education 


Roosevelt Aviation School 


Washington Preparatory 


University of California 


at Loss Angeles 


University of Glasgow - 
B. Sc. 


Techn. Hochschule 
Braunschweig 


Trinity College 
Pembroke College 


Texas Technological 
College - B.S. 


University of Tulsa - 
B.S. 


University College in 
Dublin - Diploma in 
Social Science 

College of Marketing in 
London- Diploma in 
Marketing 


Company Position 


Work Manager 
Bell Helicopter 
Company 


Managing Director 
Diamond Power 
Specialty Ltd. 


General Manager 
Industrial Furnace 
Division of 

Brown, Boveri & 
Cie, AG 


Group Technical 
Services Manager 
Imperial Tobacco 
Group Limited 


Assistant 

General Manager of 
Marketing 
Continental Oil 
Company 


Manager, Operations 
Division Computing 
Department 

Phillips Petroleum 
Company 


Chief Marketing 
Officer 
Irish Dairy Board 


Senior Executive 


Fligny, G.P. 


Flint, R.D. 


Gailey, J.S. 


Haeffner,Jr.,P.C. 


Hall, Rb ee J.N. 


LeMasters, G.E. 


Lewis, W.E. 


Loton, B.T. 


Lubben, H.E. 


APPENDIX E 


Education 


Faculty of Law in Paris - 
Bachelor 


Bordesley Green Technical 
University of Aston -B.Sc. 
University of Birmingham - 
Diploma 


University of Missouri - 
B.S. 


Williams College —- B.A. 
University of Texas - 
B.A. 


University of Kentucky 
B.S.M.E. 


Oklahoma State 
University - B.S. 


Melbourne University - 
B.Met. Eng. 


Bergakademie Clausthal - 
Engineering and Ph.D 


Profile of Participants (continued) 


Company Position 


Long Term Planning 
Rirector 
Savonneries Lever 


Production Engineer 
Joseph Lucas Limited 


Plant Manager 
Corning Glass Works 


Vice-President 
Real Estate & Mortgage 
Loan Department 

Chase Manhattan Bank 


Vice President 
Lone Star Gas Company 


Executive Assistant 
to Executive Vice 

President - Indiana 
& Michigan Electric 


Assistant General 
Manager, Room Air 
Conditioning Division 
Westinghouse Electric 
Corporation 


General Manager 
The Broken Hill 
Proprietary Coy.Ltd. 


Operations Manager 
Brigitta -— Elwerath 


APPENDIX E 


Profile of Participants (continued) 


Senior Executive Education Company Position 
Kossov, O.H. Moscow Power Institute - Head of group for R&D 
Post Doctoral Institute for Control 


and Management Problems 


Meyer, G.C.W. Swiss Federal Institute Director of Central 
of Technology - Organization of the 
Electrical Engineering group Swiss Aluminium 
Ltd. 
Petersen, W.E. Iowa State University - Vice President of 
B.S. Marketing-Residential 


Division Honeywell Inc. 


Rives, J.R. Texas Technological Vice President-Manager, 
University ~ B.S. Equipment Division 
J.M.Huber Corporation 


Sedgley, G.H. University of Toronto - Vice President, 
Bsc. Marketing Control 
Systems Group 
Honeywell Limited 


Steck, R.J. Technical University Assistant Vice 
Berlin - Ph.D President - Research 
Production & Engineer 
Henke1l-CIE-GMB H Corp. 


Tevoedjre, A. University of Toulouse - Assistant Director- 
d'enseignement plus General International 
CAPES Graduate Institute Labour Office 
of International Studies- 
post graduate diploma 
University of Fribourg - 
doctorate 


APPENDIX E 


Profile of Participants (continued) 


Senior Executive Education Company Position 
Trippe, K.A.B. University of Missouri - Assistant Treasurer 
LL.B. Corporate and 
Kansas University - B.S. International Utilities 


Financing Corporation 


Wells, J.L. Yale University —- B.A. Vice President and 
University of Virginia Secretary CIBA-GEIGY 
Law School - LL.B. Corporation 


Date 


Thursday 
30 March 


Saturday 
1 April 


Tuesday 
4 April 


Wednesday 
5 April 


Thursday 
6 April 


Friday 
7 April 


Monday 
10 April 


Tuesday 
11 April 


Wednesday 
12 April 


Thursday 
13 April 


Friday 
14 April 


Tuesday 
18 April 


Time 


7.30 
2.00 


2.00 
2.00-5 .00 
5.00 
3.30 


2.00 
2.00-5.00 
5.00 

2.00 
2.00-5.00 
5.00 
2.00 
2.00-5.00 
5.00 

2.00 
2.00-5.00 
5.00 

2.00 
2.00-5.00 
5.00 

2.00 
2.00-5.00 
5.00 

2.00 
2.00-5.00 
5.00 
2.00-5.00 


APPENDIX F 


Senior Executive 
Management Game Schedule 


p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 
p.m. 


p.m. 


Activity 


Game Manuals distributed 


Demonstration of Console Use 


Introduction to Game 


Console 


Quarter 


session 


12 data input form 


Individual goal due 


Quarter 


12 results returned 


Firm goals due 


Console 
Quarter 
Quarter 
Console 
Quarter 
Quarter 
Console 
Quarter 
Quarter 
Console 
Quarter 
Quarter 
Console 
Quarter 
Quarter 
Console 
Quarter 
Quarter 
Console 


Quarter 


session 

13 data input form 

13 results returned 
session 

14 data input form 

14 results returned 
session 

15 data input form 

15 results returned 
session 

16 data input form 

16 results returned 
session 

17 data input form 

17 results returned 
session 

18 data input form 

18 results returned 
session 


19 data input form 


Debriefing session 


APPENDIX G 
Contents of the MIT Management Game Manual 


Management Game Notes 
Brief introduction to game and performance evaluation. 


Schedule 

Detailed schedule of activities for the game. 
Overview 

Complete description of the details of the game. 
Explanation of Report Items 


Capsule summary of the meaning of items contained in the reports. 


Market History 
Summary history of the markets during quarters 2 through 11. 


Firm History 


Full financial reports covering operation of a firm in 
quarters 2 through ll. 


Console Input 
Description of data input and command use common to all systems. 


Marketing Decision System (MDS) 
Description of Marketing Decision System. 


Financial Planning System (FPS) 
Description of Financial Planning System. 


Statistical Analysis System (SAS) 
Description of Statistical Analyses System. 


Computer System Characteristics 

Description of some pertinent characteristics of the computer. 
Flow Charts 

Capsule descriptions of MDS, FPS and SAS in flow chart form. 
Goals and Decision Questionnaires 


Forms to be completed and returned to game administrator. 


Paper 
Graph and accounting paper for the convenience of the team. 


APPENDIX H 
Firm Goals 


The firm should complete this form. In order to determine 
performance relative to desired goals, we would like you to 
establish your corporate objectives. Five factors should go into 
this determination. 

Market value of stock at end of last period 
Return on net equity over last four periods 
Market share at end of last period 

Total earnings over total simulation 


Earnings per share 


Fifteen points are to be assigned to a combination of these 
five factors. Rank each factor from 5 to 1, 5 being the most important, 
1 being the least. Each factor must be assigned a unique number from 
the set (1 to 5). No equal rankings are allowed. 

Factor 


Points Assigned 


Market value of stock 


Return on equity 
Market Share 


Cumulative earnings 


Earnings per share 


10. 


ll. 


LZ 


13. 
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